engineering

ChatGPT (Generative Pre-Trained Transformer) has taken the world by storm, with Generative AI becoming the mantra of Silicon Valley. However, many people, including myself, find it challenging to truly understand this emerging field. Is it simply about enabling conversations with an all-knowing android? And how does it differ from Machine Learning, the practical application dominating the scene today? GPT is a type of Large Language Model (LLM), a category of machine learning model trained on massive text corpora....

PHR (Personal Health Records) are a modern innovation designed for interoperability. FHIR is the existing protocol that is leveraged for creating and managing health records. Ultimately, everything in FHIR is represented as a `resource`. There are a total of 157 resource types, and each of them has a baseline that can be extended by providers to stitch together their own customization over the standard protocol. Each resource has attributes, and each attribute can be of different datatypes....

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure. — Melvin E. Conway Geographically distributed software teams have become a necessity in many organizations due to cost, manpower, compliance, or growth reasons. Ensuring that these teams function well is crucial for the success of software projects. Here are some strategies to achieve this: Build a strong, shared architectural practice: Foster a culture of collaboration and knowledge sharing to ensure that communication patterns are not influenced solely by the teams’ physical locations....

I enjoy reviewing classic industry papers periodically. The DynamoDB paper is fascinating, approachable, and deepens insights on almost every read. Finally, its a good reminder that good programs accomodate different concerns to optimize for others. DynamoDB was built for handling amazon.com’s shared shopping cart problem. Most Amazon customers have a single household account, and any household member should be able to add items to a shared shopping cart at different times on their devices....

Building and operating large-scale software is a relatively young field, and many industry practitioners, including software engineers, often overlook the necessary considerations for doing it well. While “Hello World” examples of software development may serve as good tutorials, they can also give the impression that solving software problems is a trivial task, which fails to acknowledge the significant amount of work and dedication required to achieve quality results. When developing a typical service solution at a big tech company, the following factors must be taken into account:...

Tech is changing - more people are using cloud services for computing and storage. Big companies provide the infrastructure and rent it out to customers through network interfaces. The benefits of using cloud services are that customers have easy access to lots of computing power and storage. The company takes care of software updates, so customers can focus on using their software. But what if a customer already has their own data center and running workloads is cheaper there than on the big cloud vendors?...

Avoiding bad situations, and preventing failures plays second fiddle to building and releasing new features and processes. For example, there is often low motivation or even discussion on improving legal, security, and compliance processes, as they are deemed boring and complex. However, engineering teams continue to pay high cost of proving compliance or firefight incidents that could have been prevented with better telemetry and alerting. Similarly, removing brittle product features and deprecating failed products are taboo, and users continue to deal with abandoned half starts....

Skilled and talented engineering teams often become “known” for the solution they provide successfully to customers. This is dangerous. Impact to customer value Once an engineering team comes to identify themselves with their solution, they inadvertently try to fit and adapt the problem to the solution, instead of the other way around. More worryingly, teams tend to discount better alternate solutions, even if they fit the problem better. Reasons given are higher amount of time to solve, unknown unknowns, low skills etc....

Software as a Service (SaaS) business is unlike .COM, which provided internet based alternatives for conducting commerce (example: Amazon), auctions (example: eBay), travel (example: Expedia), and finance (example: E*TRADE) using desktop browsers initially, and now using mobile applications. SaaS is a paradigm shift in the implementation, delivery and usage of software applications. SaaS uses two ways to distribute software applications: Web apps: use the public internet to deliver software to users, making delivery costs close to zero Mobile apps: use proprietary app stores, which slap on extra penalty of 30%....

Setting up devops is a learning journey for most programmers. Early in my engineering career, before cloud computing was invented, making my programs work in my company’s datacenter could be summed up as making the program: configurable: by having no hardcoded paths and secrets in the code, supporting feature flags (both static - which were driven through property files, and dynamic - driven through databases) deployable: in different machines other than mine....