Powertools for AWS Lambda (TypeScript) v3 - Ideas and notes

[dreamorosi]

I would like to open a discussion to gather and ideally discuss ideas and changes for our next v3 milestone.

Note that at the time of writing we don't have any date nor plans for our next major version, however I think it'd be good to start gathering notes & ideas for things we might encounter along the way that we wish we could address but can't due to major refactors or breaking changes.

[dreamorosi]

Relating to the Batch Processor utility, the components of the utility follow an inheritance pattern with some abstract classes and methods. During the initial implementation we weren't necessarily the most diligent and so almost all methods were marked as public.

Some of these methods (my feeling is that it's most of them) could potentially be converted to protected instead, since they are meant to be used only by classes extending the base class.

As part of v3 we could consider doing an exercise of minor refactoring and clean up the public API.

[HaaLeo]

Maybe consider #749 for v3 if that would cause any breaking changes.

[dreamorosi]

In aws-powertools/powertools-lambda-python#4907 the extractDataFromEnvelope() function part of the JMESPath utility was renamed to simply query(), we should do the same.

[dreamorosi]

As discussed in #3085, in v3 we will switch the experimentalDecorators setting in the tsconfig.json file to false.

In v5.0 of TypeScript, the default changed to Stage 3 decorators, which modify the signature of the decorator significantly as discussed more in depth here.

While doing this, we should also make our decorators compatible with asynchronous methods only. Today they support both but in reality they implicitly convert sync methods to async under the hood. This is not a huge issue when decorating the main function handler since AWS Lambda will handle the promise correctly, but when decorating arbitrary methods this can cause subtle bugs.

[codyfrisch]

As discussed in #3085, in v3 we will switch the experimentalDecorators setting in the tsconfig.json file to false.

In v5.0 of TypeScript, the default changed to Stage 3 decorators, which modify the signature of the decorator significantly as discussed more in depth here.

While doing this, we should also make our decorators compatible with asynchronous methods only. Today they support both but in reality they implicitly convert sync methods to async under the hood. This is not a huge issue when decorating the main function handler since AWS Lambda will handle the promise correctly, but when decorating arbitrary methods this can cause subtle bugs.

As long as this has no impacts on projects that still depend on the old experimental decorators - inversify comes to mind. Just something to consider. (They depend on parameter decorators which aren't in the stage 3 spec). I don't know if offering both is even viable.

[dreamorosi]

I have yet to do an in depth PoC, but so far I'm with you on it not being viable.

In terms of impact, we're leaving this for a major version on purpose since it will probably result in a breaking change.

For what it's worth, we don't yet have any concrete plans for v3 and I don't expect us to start working on it before end of the year at the very earliest, and anyway after having cleared our current roadmap for the year.

As soon as v3 plans are more concrete I'll publish announcements in the repo and other channels to communicate changes like this one well in advance like we did for v2, so that customers can prepare. Even after v3 is released, we will still support the current version for 6 extra months according to our versioning policy.

This is to say that there's still a ton of time before this becomes a problem, however I think it's also important to stay up to date with the ecosystem and acknowledge that there's demand for moving to the new implementation which at least for now looks like it's going to be the final one.

[dreamorosi]

Following the implementation of #3160 that adds async processing to SQS FIFO queues, to avoid breaking changes we had to invert the convention of XXProcessor being async and XXProcessorSync being the synchronous opt-in. In this case the existing SQS FIFO processor stayed sync and we added an asynchronous one.

At minimum, on v3 we should standardize this implementation to follow the rest of the Batch Processor classes.

Additionally, as @am29d pointed out, there might be an opportunity to streamline the implementation and API of this utility significantly while making minimal breaking changes.

For example, we could explore reducing the number of classes to have only one batch processor for event source, and then have both synchronous and asynchronous methods on it, so that customers don't have to think which one they need to import and use but just call BatchProcessor.process() (async) or BatchProcessor.processSync().

We are unsure if this is possible or the extent of breaking changes it'll require - for now we're adding this to the v3 list to explore further.

[dreamorosi]

Another item we're looking to remove or rework is the custom configuration services from the core utilities.

Today it's possible to pass a custom configuration service to the constructor of Tracer, Metrics, and Logger which will be used during initialization.

Currently this feature doesn't seem to have gotten any traction, most likely also because it's not documented anywhere, but also because it's too rigid.

Defining your custom configurations service requires you to reimplement the entire interface, which might not be desirable when you want to only override one behavior.

[dreamorosi]

In v3 we should stop including the service dimension to metrics emitted when the service name is not specified via POWERTOOLS_SERVICE_NAME env variable or serviceName constructor options.

Currently we still add the dimension but with value service_undefined, which adds no value and only increases the size of the log emitted.

Customers have asked us to change this but doing so requires a breaking change, so I'm adding this here.

More info in #3374.

[am29d]

Change POWERTOOLS_TRACE_ENABLED to POWERTOOLS_TRACE_DISABLED to make it consistent across all runtimes.

[dreamorosi]

As a customer, when retrieving a SSM parameter stored as StringList and applying the transform: 'json' I want to get back an array of strings rather than a string like this foo,bar.

Currently we do the latter, but we should consider doing the former even though the StringList is not a proper JSON string.

More info in #2041.

[dreamorosi]

Following the discussion in #3781, we could consider changing how the partition and sort key are created in Idempotency.

When using composite keys, our current implementation sets the keys like this:

• PK: idempotency (or other static value)
• SK: function-name#hash-value (the hash is derived from the payload)

This is done to allow us to read/write items while knowing only the payload and thus minimize reads.

The problem with this design - which again, affects only customers using composite keys - is that it creates a hot partition due to the static nature of the PK.

For customers who are seeing hot partition issues and are also sharing the same DynamoDB table for multiple idempotent functions, the issue can be partially be mitigated by using the keyPrefix option documented here

If instead hot partition becomes an issue with a single function - then there's nothing else to do other than extend the DynamoDBPersistenceLayer class and update the getKey() method to create a different PK value.

This entire experience is not optimal, and the discussion in the linked issue highlighted that there might be a better way, which consists in writing the keys like this (the payload hash is used in both keys):

• PK idempotency#hash-value
  SK: function-name#hash-value

This is a clever design that we did not think of, and we should make extensive tests before deciding to implement it. However given that it's a breaking change in how the keys are generated, we're putting this on hold until our next major version to reconsider.