Important aspects of the development context A list of some of the different ways in which your context might be different from that of other people you might be interacting with on the internet for learning how to build technology solutions
- Size of code base.
- Size of development team.
- Nature of hierarchies within the organisation running your software development.
- Business domain, e.g. medicine, law, finance, physics, gaming etc.
- Speed at which you must react to market changes. Especially, the speed at which you might need to scale operations, or develop new features.
- Need for or use of specialists vs generalists e.g. for web development, you have “full stack” developers at one end of the scale, vs specialists in databases, backend application code (which may have multiple layers), frontend application code, design, UI/UX etc.
- Priority given to avoiding or fixing faults. Consider critical medical systems, for example, compared with an experimental game you are developing for fun in public.
- Priority of getting the best possible performance. Performance could be measured in terms of memory usage, CPU time, network latency, network bandwidth etc. For some projects, a 1% improvement is considered a very worthy goal, for others, solutions that are thousands of times slower than optimal could be absolutely fine, and a factor of 10 improvement might not even be worth it.
- Kind of users your software has. They could be other developers like you, or end users with a massive range of ability with computers.
- Number of users of your software – this could easily range over 9 orders of magnitude.
- Amount of data you process – again, a massive range.
- Relative value to your “business” of each customer.
- To what degree you know what hardware your software will run on.
- How close you are to hardware limits (compare embedded systems to most desktop software, for example).
- How many different environments (e.g. operating systems) your software needs to work in.
- Extensibility requirements – from “none”, through “extendable by developers” or “extendable by end users” (for example with embedded scripting engines)
- Long term maintenance needs. Some projects are kind of “throw-away”, and some see few changes after a “release” e.g. some (but not all) games. On the other hand, some software will be maintained for decades, sometimes being heavily changed after initial development, other less so.
- Need for new developers and maintainers. In some projects, you’ll have a pretty static team, others will see huge amount of turnover.
- Ability to find more developers and maintainers. Some projects have no money at all, while for others money is no object. Some may attract new developers by virtue of being “fun”, others may not be able to.
- Backwards compatibility needs with external dependencies. Differences here will make a large difference in how much design you need to do up front.
The ultimate guide to developer counter-productivity
- Reactive, unplanned work
- Context switching and startup costs
- Non-value-adding administrative & compliance work
- Ineffective planning
- Dependency management overhead
- Ineffective meetings and communication
- Redundant manager briefing & orientation
- Consensus seeking and decision-making drag
- Ineffective collaboration arrangements
- Lack of help and support
- Ineffective documentation
- Poorly maintained code and docs
- Workarounds
- Inefficient onboarding
- Tooling challenges
- Manual, automation-friendly tasks
- Building low-value features
- Unreleased code
- Artificial deadlines & rushing
- Poor timing/energy fit
- Skill mismatch
- Knowledge loss
- Missed opportunity costs
- Second and third-order effects
- Broken Authentication:
- Authentication mechanisms are often implemented incorrectly, allowing attackers to compromise authentication tokens or exploit implementation flaws to assume other users’ identities temporarily or permanently.
- This compromises API security overall.
- Unrestricted Resource Consumption:
- Satisfying API requests requires resources such as network bandwidth, CPU, memory, and storage.
- Successful attacks can lead to Denial of Service or an increase in operational costs.
- Broken Function Level Authorization:
- Complex access control policies with different hierarchies, groups, and roles can lead to authorization flaws.
- Attackers can exploit these issues to gain access to other users’ resources or administrative functions.
- Unrestricted Access to Sensitive Business Flows:
- APIs vulnerable to this risk expose business flows without considering how excessive usage could harm the business.
- This doesn’t necessarily come from implementation bugs.
- Security Misconfiguration:
- APIs and their supporting systems often have complex configurations that can be missed or not follow security best practices, opening the door for various types of attacks.
- Improper Inventory Management:
- APIs expose more endpoints than traditional web applications, making proper and updated documentation crucial.
- Maintaining an inventory of hosts and deployed API versions is also important to mitigate issues such as deprecated API versions and exposed debug endpoints.
- Unsafe Consumption of APIs:
- Developers tend to trust data received from third-party APIs more than user input, leading to weaker security standards.
- Attackers may target integrated third-party services instead of directly compromising the target API.
Offline is just online with extreme latency
- Don’t think of offline and online as two different things, because then you create software that splits experiences into two different modes: 1) “online” which gets all the support and attention, and 2) “offline” which gets little to no thought at all.
- This kind of idea would move you away from a product full of API calls to one based on data synchronisation.
- When you shift the way you think about building things you open new doors to building different things. So, even if you don’t like the idea of data synchronisation over API transactions, it’s at least worth entertaining and imagining what’s possible in a paradigm that’s different than what you’re used to.
- You even begin to realise that “offline” has its own gradations: latency of seconds, minutes, hours, days, weeks, or more! They’re not all the same and represent a more accurate, all-encompassing picture of the kinds of environments real-world users live in.
Some reasons why Instacart was spending 50+ million on Snowflake
- Oversized warehouses being kept active by spaced out queries. The warehouses can be downsized or queries clustered together to reduce idle time
- Poorly optimised data - clustering that hasn’t been followed up on, data being ingested and not cleaned up before being queried and thus not.making use of clustering well
- Bugged third party tools hammering Snowflake with queries
- Heavy dashboard views being created by analysts without proper consideration for performance
Others
Substack edition
Important aspects of the development context
A list of some of the different ways in which your context might be different from that of other people you might be interacting with on the internet for learning how to build technology solutions
Size of code base.
Size of development team.
Nature of hierarchies within the organisation running your software development.
Business domain, e.g. medicine, law, finance, physics, gaming etc.
Speed at which you must react to market changes. Especially, the speed at which you might need to scale operations, or develop new features.
Need for or use of specialists vs generalists e.g. for web development, you have “full stack” developers at one end of the scale, vs specialists in databases, backend application code (which may have multiple layers), frontend application code, design, UI/UX etc.
Priority given to avoiding or fixing faults. Consider critical medical systems, for example, compared with an experimental game you are developing for fun in public.
Priority of getting the best possible performance. Performance could be measured in terms of memory usage, CPU time, network latency, network bandwidth etc. For some projects, a 1% improvement is considered a very worthy goal, for others, solutions that are thousands of times slower than optimal could be absolutely fine, and a factor of 10 improvement might not even be worth it.
Kind of users your software has. They could be other developers like you, or end users with a massive range of ability with computers.
Number of users of your software – this could easily range over 9 orders of magnitude.
Amount of data you process – again, a massive range.
Relative value to your “business” of each customer.
To what degree you know what hardware your software will run on.
How close you are to hardware limits (compare embedded systems to most desktop software, for example).
How many different environments (e.g. operating systems) your software needs to work in.
Extensibility requirements – from “none”, through “extendable by developers” or “extendable by end users” (for example with embedded scripting engines)
Long term maintenance needs. Some projects are kind of “throw-away”, and some see few changes after a “release” e.g. some (but not all) games. On the other hand, some software will be maintained for decades, sometimes being heavily changed after initial development, other less so.
Need for new developers and maintainers. In some projects, you’ll have a pretty static team, others will see huge amount of turnover.
Ability to find more developers and maintainers. Some projects have no money at all, while for others money is no object. Some may attract new developers by virtue of being “fun”, others may not be able to.
Backwards compatibility needs with external dependencies. Differences here will make a large difference in how much design you need to do up front.
The ultimate guide to developer counter-productivity
Reactive, unplanned work
Context switching and startup costs
Non-value-adding administrative & compliance work
Ineffective planning
Dependency management overhead
Ineffective meetings and communication
Redundant manager briefing & orientation
Consensus seeking and decision-making drag
Ineffective collaboration arrangements
Lack of help and support
Ineffective documentation
Poorly maintained code and docs
Workarounds
Inefficient onboarding
Tooling challenges
Manual, automation-friendly tasks
Building low-value features
Unreleased code
Artificial deadlines & rushing
Poor timing/energy fit
Skill mismatch
Knowledge loss
Missed opportunity costs
Second and third-order effects
OWASP security risks in 2023
Broken Authentication:
Authentication mechanisms are often implemented incorrectly, allowing attackers to compromise authentication tokens or exploit implementation flaws to assume other users' identities temporarily or permanently.
This compromises API security overall.
Unrestricted Resource Consumption:
Satisfying API requests requires resources such as network bandwidth, CPU, memory, and storage.
Successful attacks can lead to Denial of Service or an increase in operational costs.
Broken Function Level Authorization:
Complex access control policies with different hierarchies, groups, and roles can lead to authorization flaws.
Attackers can exploit these issues to gain access to other users' resources or administrative functions.
Unrestricted Access to Sensitive Business Flows:
APIs vulnerable to this risk expose business flows without considering how excessive usage could harm the business.
This doesn't necessarily come from implementation bugs.
Security Misconfiguration:
APIs and their supporting systems often have complex configurations that can be missed or not follow security best practices, opening the door for various types of attacks.
Improper Inventory Management:
APIs expose more endpoints than traditional web applications, making proper and updated documentation crucial.
Maintaining an inventory of hosts and deployed API versions is also important to mitigate issues such as deprecated API versions and exposed debug endpoints.
Unsafe Consumption of APIs:
Developers tend to trust data received from third-party APIs more than user input, leading to weaker security standards.
Attackers may target integrated third-party services instead of directly compromising the target API.
Offline is just online with extreme latency
Don't think of offline and online as two different things, because then you create software that splits experiences into two different modes: 1) “online” which gets all the support and attention, and 2) “offline” which gets little to no thought at all.
This kind of idea would move you away from a product full of API calls to one based on data synchronisation.
When you shift the way you think about building things you open new doors to building different things. So, even if you don’t like the idea of data synchronisation over API transactions, it’s at least worth entertaining and imagining what’s possible in a paradigm that’s different than what you’re used to.
You even begin to realise that “offline” has its own gradations: latency of seconds, minutes, hours, days, weeks, or more! They’re not all the same and represent a more accurate, all-encompassing picture of the kinds of environments real-world users live in.
Some reasons why Instacart was spending 50+ million on Snowflake
Oversized warehouses being kept active by spaced out queries. The warehouses can be downsized or queries clustered together to reduce idle time
Poorly optimised data - clustering that hasn't been followed up on, data being ingested and not cleaned up before being queried and thus not.making use of clustering well
Bugged third party tools hammering Snowflake with queries
Heavy dashboard views being created by analysts without proper consideration for performance