The Bank of New York Mellon Corporation (BNY) Earnings Call Transcript & Summary

Hello, everyone, and welcome to day 2 of Camunda in Action. My name is Shane, and I am thrilled to be your moderator today for this track, where we're going to be exploring real-world use cases of process orchestration, told by the experts who use these tools every day. Just like yesterday, we're going to be using Slido to capture your audience questions. So please, if you haven't already, scan the QR code or visit Slido.com and you can put CC Stage 2 at the top of the page. And for all of our virtual attendees on the right-hand side, there should be a little tab that says Slido. So please use that to pepper the presentation today with as many questions as we can for Michael because we really appreciate that. Without further ado, I'd like to welcome Bank of New York Mellon's, Michael Goldverg. He's a Managing Director and distinguished engineer there. And today, he's going to share how his... [Audio Gap]

Thank you, everybody. Can you hear me well? Great. Okay. First, introductions. I'm Michael Goldverg. I work for Bank of New York. I do software development for living. So I'm an engineer. All the other titles are not as important, especially in the context of where we are, but I do write software a lot. And I've been doing it all my life. Unfortunately, spent most of my time at the financial industry and still happy about doing it. So I haven't had a chance to learn other industries. All my career has been firmly with finance. So I'm also an agile practitioner, so I coach and practice agile product development myself. And lastly, I'm do it yourself in 2 years. So I like doing things with my own hands. So I tie all the floors in the bathrooms. I lay the hardwood flooring. I'd like to paint. So I do all the things myself. I try to. Sometimes it works, sometimes it doesn't. But generally, my wife gives me thumbs up with results of my work. So -- and I do the same with software development. So everything that I try doing with my own hands. Okay. So let's talk about the problem first. About 5 years ago, we were asked to solve a very complex problem in the bank. Many people, thousands of people doing the work manually across multiple groups and performed fairly complex coordination activities every day. So many groups depend on each other, and they used to coordinate their work by sending e-mails to each other and literally thousands of e-mails were flying between those groups on a daily basis. So one group of people would complete their set of work for a specific set of accounts, pass the work to the next group of people, and they would have to do something. And very often, the second group would find problems that they'll have to communicate back to the first group of people that they have to do something and so on. So you get the picture. Thousands of people trying to coordinate work on a daily basis between each other, very often leading to time runouts, mistakes and so on. What complicates the picture even more is the fact that we operate with very strict deadlines on a daily basis. So every group that works within the bank has a set of service level agreements or deadlines they have to meet every day. So if they're not meeting the deadline, sometimes it leads to financial penalties or other escalations to the clients. So a very complex problem. And many times, we've tried to solve it in the past, and those attempts didn't yield any decent results. So 5 years ago, we were asked to try again. And we had to learn from the prior experiences and do something different. So one of the things that we've decided to do -- if we go to the next slide. One of the things that we decided to do was we tried to model the business process ones. Just to give you an idea what was happening before, the attempt to automate the business process was not successful. Part of the reason why was because many business processes were modeled differently. So we ended up with hundreds of variations of the business process. And every account that had to be supported by this business process was modeled differently or group of accounts rather. What ended up happening was that every time the changes required, many business models have really changed, and it was very difficult to support and eventually the process ended up being [ stopped ]. So as opposed to that, we chose to implement a single supermodel, we call it supermodel, a base model. And that model had to absorb all the variability of various accounts in order to support the business process. So we had to introduce things like conditional tasks. Something that Camunda unfortunately doesn't support today, but I'm already in conversations with Daniel. Hopefully, they will one day. So imagine a scenario where for a set of accounts, a task has to be implemented, but for other groups wouldn't. So one of the things that we've implemented is the support for [ toggleable ] tasks. So one task can be enabled for one group, but not enabled for another. So that was one of the key things that we had to consider to support single model across various groups of accounts. The second thing, which was equally important, was the reuse of the various sub models that we ended up modeling. So I'll give you just an example of those. So you could see various reusable constructs that we had to bake into our main process in order to make it supportable across all the accounts. So top left, you see a very basic construct where a step or a task is a message received task, which is waiting once it's entered state for the for the external confirmation with a timer that allows us to exit this process if certain time has passed. So -- and in many elements of the main process, we support this concept, where the steps are terminated by the timers. So we could move forward if we pass a certain time frame. So the second, at the bottom left and right, are the 2 components of the manual tasks that we support in the process where the tasks are defined for a specific account, and those tasks are modeled either is make a [ checker ], which you see on the right. And we have a repeating step that they are modeled into or just the checker tasks, where we have a single set of tasks that are performed by individuals. So they make a checker model that you're probably aware is implemented in such a way where the person who performed the maker activity cannot be performed the checker activity and therefore is excluded from authorization. So these are the types of things that we found throughout the modeling exercises that we are using in the model. But let me step back and just give you a bit of a color. So this is a 4-, almost 5-year long journey. So we didn't start with such a comprehensive model that you see on the right. I just put it here just as [ carrier ]. So we started with something a lot simpler. So we met with the business, and we had a set of conversations with them about what they need to do, how they do their work today. And every step in the first implementation of the model was a manual task. Very simply how you heard in the keynotes today that's what we've done. Every step was manual task, nothing more. We didn't automate anything. We just tried to orchestrate what people were doing on a daily basis. And over time, we started evolving the model where some of the tasks would be replaced with their automated equivalents. Some of the task would be further explored into subprocesses and so forth. So after the 4.5 years, we ended up with a much more comprehensive and complex model that supports the business today. But it started with a simple picture. The second part of the story that I wanted to share with you is the concept of the blue-green model. So it was another interesting problem that we had to solve over the years, how do we reliably deploy the changes in software without taking too much time to do that and without breaking anything? So I'm sure you've heard about blue-green model. Martin Fowler introduced it many years ago. So we tried to apply this concept to Camunda base deployment. The challenge is that, at least in our case, that we're running against a single database. So imagine, we have multiple clusters, primary cluster where the users use the system. And then we have a secondary and maybe even more clusters that are available to us to do the deployment. So the first problem is how do you make the software run on the secondary cluster without interfering with the primary system, while connecting to the same database. So for those of you who know how the BPMN engine works, once it connects to the database, it starts performing the tasks. So the job executor starts performing the operations, and that's not what we want to see. So we wanted to see that the passive cluster, while deployed with a potentially different version of the software, still operates without touching the main system. So the users could continue reliably using the system. In order to make this happen, we had to introduce the toggle, basically a control, which tells the cluster that it's not an active cluster. And we had to train Camunda through the tricks and the plug-in to not perform the operations unless the cluster is an active cluster. So that was the first thing that we had to do. So the active cluster will be performing transactions, while the secondary cluster, the passive cluster, would not. It would simply start and not perform any activities on the Camunda database. So the second thing -- the second challenge was related to the versions of the models. So imagine we deploy version 1, and it's running on an active cluster. And now we're deploying the software to the secondary cluster, that's not active. And we deploy a newer version with potentially different variations of the BPMN schemas. So now the question is, how can the process be tied to specific versions of the models and not interfere with the others? So that was another thing that we had to do. We had to change our software in such a way, where each version of each release would only recognize its own set of models and ignore all the others that are deployed into the repository. So while repository allows multiple versions of the models to be deployed, you need to make sure that the version of the software that you're running only can recognize the model that you wanted to work with, okay? So that was the second thing that we had to do. And we had to do it at 2 levels. First, when we programmatically start the process instances, we had to resolve at the run time, which version of the software -- sorry, which version of the model we would use to start the process. The second is when you start activities within the process itself, collectivity steps. In those cases, we also had to, instead of hard coding the version, we always operate with the latest version of the model. We had to choose which one to use. So we built a small service that allows us to resolve the version of the model that we should use at run time. And the reason we had to do it is because the version of the model is not even known until we deploy them. So imagine when we're developing the software packaging, the application to a [ jar ], let's say, right? So we have the models in the jar, but we don't know which version they will have in the run time environment. You don't know this until you deploy those models into the live system. So that's why we had to make this process dynamic. So at run time, the service would determine which version of the software, which version of the model, sorry, to use when we start the process, okay? So that was the second thing. The third thing, testing. So we've deployed the software to the second cluster. Now we want to validate that the set of second cluster is operational. How do we do it without starting the Camunda, without starting the BPMN engine because, as I said before, if the cluster is passive, BPMN engine doesn't do anything. So we have to make it work. So we introduced the third stage, we called, testing stage. And during the testing stage, the BPMN engine would start. But how do we start it without interfering with production? We have to introduce a secondary database schema. So the engine that starts in a testing mode does not connect to the production database. It connects to the secondary schema, which is empty, doesn't have anything, except the new models that are deployed, and it allowed us to perform all the necessary validations in the secondary cluster, while the primary cluster is to use for production. So the secondary schema was critical for us to be able to reliably validate the software as part of the deployment without interfering with the production. And the last piece is process migration. So now we've tested the software. It's running in the secondary cluster. Now it's easy enough to switch the cluster, declare one active, declare the second one passive. Viola, we have the software running. What do we do with the processes that are in progress? So we had to implement the live process migration of any process that is running on the older version, if we determine that the process migration is required. So the termination is a complex part because sometimes we make the changes to the BPMN models that are backward compatible that do not require necessarily to migrate the process. But in some cases, those processes are not comparable. For example, you change the way you interact with the delegated code or you introduced a new step or remove the step that used to be supported. So in those cases, you need to migrate the processes. So we've implemented an automated process migration that allows the system to recognize that the existing processes that are live running in the system are running against outdated version of the model and automatically perform those migrations. So these are the 4 steps that we have to do to support the blue-green model. So I'll repeat myself, just to summarize. So the first thing is to introduce the state of the cluster between active, passive and testing. Testing is important because without it, you cannot validate the software that you released. The second, we have to implement support for multiple versions in the model, so they don't conflict with each other. So the version of the software that's running can only recognize and work with their own versions of the BPMN models. The third one is secondary database, secondary schema that allows you to run BPMN engine against it without interfering with production. And finally, the process migration, migration of the live process if the schema is not compatible with each other. So these are the 4 things that we have to consider in order to make the blue-green deployment model with Camunda work and work well. Okay. The next thing, optimistic locking exceptions. I'm sure you all heard and experienced the horror of dealing with them. So we are not unique. I can't claim that we've beaten them. This evil will not go away, and at least until version 8 is live and proven that optimistic locking exceptions don't exist anymore. But we found the ways to deal with them and considerably minimize their occurrence. So there is no super trick that I could offer you. But there are a few things I can share with you what we have to do because we're running the BPMN engine on multiple instances in the cluster. So we scaled the system beyond what we can do with a single installation with a single running BPMN engine. So we had to do something that normally, people wouldn't do. We had to introduce external locking system. So just step back to talk about how optimistic locking exceptions work and why do they happen? They usually happen for 1 of 2 reasons. First reason, you try to update an entity, a database entity from token current processes, optimistic locking exception. Or the second, you perform multiple operations in parallel on the same process instance. And that is quite common when you have any parallel gateways, you start running the processes in parallel. And sometimes they start stepping on each other. Within a single engine, this problem is easily solved because you can delegate work to job executor and job executor normally with perform operations one after another. It would recognize that you are trying to update the entries from multiple processes, and it would resolve the problem for you. And by the way, one of the recommendations that I will be giving you later on is to try to leverage job executor more aggressively. But when you run the process on multiple instances of BPMN engine, job executor doesn't help anymore because it only operates within a single instance. So we have to introduce an external locking mechanism. How does that work? We've implemented a Hazelcast map, where every process that starts registers itself as an entry. So we have 10,000 processes. We have 10,000 entries, where the key is the process [ instance ID ], okay? So now every time a process tries to perform any update operation on the task, let's say, closing a task or claiming a task, right? We're first trying to acquire a lot on the entry corresponding to that process [ instance ID ], okay? And the process will wait until it can lock that entry in Hazelcast map before it can perform any other operations. So that allowed us to greatly reduce the number of optimistic locking exceptions across our platform. Hazelcast is a fairly high performing in-memory data grid. It allows us to perform those activities very quickly. So the overhead is negligible. It's even impossible to notice it. It's microsecond, not in milliseconds. So if it's well tuned, you don't need to worry about this external locking introducing the overhead in your system. But it does the trick. It allows you to pause the process and wait for the lock to be available before it can continue and thus, avoiding many of the optimistic locking exceptions. So the second thing we did was we've added an interceptor that allows us to accumulate a set of activities before flushing them if they're not transactional. So if the transaction fails and rolls back, we don't perform those activities. If it succeeds and committed, then those non-transactional activities can be executed after the [ fact ]. So that was another trick that we employed to try to avoid the situation where optimistic locking exception happens within the transaction and some of the external updates or interactions or invocations already took place, while the transactions rolled back and we're back to where we started. So the delegation of that code and delaying of that code until the transaction is fully committed by Camunda is another trick that allows you to avoid inconsistencies in optimistic locking exceptions sometimes cause. The third thing is the reentrance of the code. So this is more of a wish list and a general guideline. So in order to fight the optimistic locking exceptions, sometimes you will choose to make the step in the process synchronous through declaration of a synchronous continuations. And by doing that, you're effectively delegating the work to the job executor. And one of the nice features of the job executor, you can tell it to retry failed transaction a number of times with the delays. So you could say I want to delay the process by 2 seconds and try it one more time, right? Simply speaking. The problem is that if your delegated code is not re-entered, you may introduce problems with your external systems. Let's say, you're sending a message, and your external system cannot tolerate same message being sent more than once. So in this case, you can't really employ the retry logic because it would mean that you would be resending the same message over and over again. So this is just a recommendation when you have to deal with optimistic looking exceptions by enabling synchronous continuations, make sure that the code that's triggered by the step is reentered that can be performed over and over again without any harm to the system. So the general problem and challenge with retries is that sometimes it could slow down the system. So if you are not careful with performing retries, especially when you have heavy transactions, you could put the system out of commission by simply causing too many retries, when their head activities going on. So use retry logic with care and avoid complex and slow transactions with the retry logic that could damage the system, okay? Okay. So last piece of my talk today is about how to make the system run well under heavy load. And let me qualify heavy load. We're running about 50,000 to 100,000 processes every day. In our historical database, we have between 100 million and 150 million manual tasks, close to 200 million variables. So we put Camunda under heavy stress, under heavy load, every single day. Thousands and thousands of processes are running concurrently, okay? How do we make it work and work reliably? There are a number of tricks that we have to do. First is we have to keep the historical database clean. Meaning we've negotiated with our business partners, how much history we want to keep on the system. And everything beyond that, we remove aggressively. So we do not allow the database to grow beyond the allowable space, okay? So 60 days is our current mark. Anything older than 60 days we'll remove. Now we have our regulatory obligations that we have to have the data archived and stored for a long period of time, years, tens of years. So what we're doing? We are generating archived report. Before removing the data from the system, the archival process construct a fully defined report of every task, every activity with every attribute from Camunda generates the support in the form of PDF or JSON, and we publish it into the document management system as a document. That allows us to later discover if there is a question about what happened on specific day with this particular account, we can find the corresponding document interrogated and provide the answer. That includes, by the way, comments that you just capture when they close the tasks and even attachments that people [ display ]. We use business key to group multiple processes together. So for example, an account and the date would form a business key and then all the processes, all the tasks, all the activities related to the business key are bundled together into a single document and published that allows us easy discovery of those files later on. So the archival process is critical to keeping the system healthy and not growing too much. And that kept the performance of the system fairly consistent even though the volumes are quite high. The second thing is the [ querying ] of the database. What we've noticed is that few heavy or not carefully designed queries could bring the system down by simply slowing down the database to the point where Camunda starts choking. So that's another recommendation, be very careful with the queries as you design them if you have to [ query ] Camunda directly. Now in most cases, when you use Camunda API, usually the performance of the processes is tolerable and acceptable and working well. But sometimes when you have to [ query ] the database directly, that's why you have to be very careful. The job executor is the key element of performing BPMN engine. And therefore, you have to be very careful about how you tune job executer with regards to the threads that are available to it in every engine, with regards to the size of the queue that you want to have available and with regards to the number of database connections that are available on each engine, that the job executor can use to transact the Camunda transactions. So for example, you could define the set of threads to be available to job executor, you could define the maximum set of threads and you could decide on how long you're going to have a queue that will allow you to store pending requests before job executor can perform them. And there is no right or wrong answer here, but just be mindful that if you have thousands and thousands of jobs that the job executor has to perform, on one hand you could reduce the size of the queue to a bare minimum and allow job executor to perform the operations based on what's available on the job table, or you could optimize the queuing of those jobs by allowing job executor to take a batch, let's say, define a batch of 1,000 jobs. That will greatly speed up the job execution. However, if that engine goes down, the jobs are gone. They disappeared now, so you can't recover anymore. So that's the choice that you have to make, whether you want recoverability of the failed the execution engines or you want performance of the job executor. And there is no right or wrong answer, you have to make those decisions. But what you have to understand is the job executor has those abilities and you can leverage them. Another thing to keep in mind is that there is a difference between the number of threads that job executor can employ and a number of maximum number of threads that you can allow job executor to use, and those are not the same things. So job executor will not exceed the first set of threads until it starts draining the queue. Once the queue is drained, only then it will try to increase the number of threads. And that's something to keep in mind. So if you look, for example, in the console, like Spring Boot Admin console, you will see job executor using a set of threads that are below maximum when you start asking yourself a question, why is that? Why do I have the entries in the queue? But the maximum number of threads is not utilized. And this is because of the way the threading works. So it's something to keep in mind. So as your application grows and you're seeing that your engines can no longer handle the volume that you're throwing at it, it's time to increase the number of engines in your system. The reason we were able to do safely is because we started with 2. We knew that our system will eventually grow. So we knew that we would never be able to run a single engine. So we started with 2 even though we could operate with 1. Why did we start with 2? Because we had to work out all the issues related to 2 engines running in parallel, such as locking, right? So if you have an idea that your system is going to grow beyond a single engine, don't start with 1. It's much harder to work out all those issues when you're live in production and your system underperforming, then when you're just designing and experimenting. So we started with 2 for that reason. And finally, as I mentioned, we had to spend a considerable amount of time training and tuning our job executor to make sure it performs. Job executor, as I said, the heart of the BPMN engine. It's important that it's healthy and performing well, and the balance between the number of threads and the database connections that you allocate from the connection pool to the job executor is also critical. Because if job executor finds itself where it doesn't have enough database connections to work with, it will start slowing down, waiting for the connections to become available. So the rule of thumb is to allocate 1 daily base connection per thread, so you have enough. You don't have to worry about it. And the job executor can perform safely. And finally, database hardware. So we've had a couple of cases in the past 5 years, where we would find out that our database just can't handle the load that we're putting on it. So while it wasn't our primary responsibility to observe and monitor the database environment, we learn the hard way that if we don't pay attention to it, we may be in a situation where our system cannot perform as fast as we want to. So we have to upgrade the database, and that's one of the recommendations I want to leave you with. If you are facing the performance challenge in your Camunda engine, one of the things that you may need to look at is your database. I think this is all I wanted to tell you today, folks. So happy to answer any questions. And thank you very much for listening.

Well, thank you, Michael. That was super insightful. And I'm really loving the supermodel term. Like, I don't know I'm going to try to weave that into some other conversation later. I really like that idea. Again, please use Slido, everyone, both attending online as well as here in the audience if you have additional questions, but you can also upvote questions that you want answers from Michael. For us, it's a nice way to engage with them. So let's kick it off here with this first one here. How often do you change BPMN models? And how do you handle the in-progress processes when the new versions of models are deployed?

We deploy the software every week. Sometimes often more often than a week, sometimes twice a week, sometimes 3x a week, whatever our users really need. And very often, the models change. So we don't normally restrict the changes to the models. And as I mentioned, we have to perform the migrations every time the models are not compatible with the prior versions. But the short answer to the story, to this question is that we deploy the software many times a week, many times a month, several times a week usually. So the models changes all the time. And that's part of our CI/CD process where we validate everything. We make sure that the processes that validate the software perform successfully. They're green. And if the test is green, we deploy. We deploy the software. So...

So thank you, my favorite term. How long did it take to create and finalize the supermodel?

Well, 4.5 years. And we're still working on it. We haven't finalized it yet. It's an evolving model. It will continue evolving until the project is no longer valid, and we'll stop. So we don't define the target state. We change the models whenever our business users want us to change the model.

Well, that continuously improving, right? Okay. How many change requests do you receive in a month on average? I know you kind of just touched on that, but curious if you can elaborate a little bit more on the numbers there?

I would say we handle between 10 to 22 requests a week. So usually...

Are they pretty significant or...

Well, we have a team of 12 developers. So we try to release software at least once a week, and they are usually dozen to 2 dozen changes that we implement, large or small. If they are too large, we split them into smaller chunks and deliver incrementally.

Makes sense. I love the agile approach to it. Okay. How have you handled inter process communication within your environment?

Well, we use messaging. We like the synchronous communications. We use them as much as possible. It allows us to manage the process with a set of -- for the limited set of resources available. We use Kafka. We use Hazelcast queues, whatever works really, but queuing is probably the most popular way of communicating between the processes. We have scheduled tasks within Camunda engine that usually either receive those messages and then notify the specific process instances or we do something similar when we don't have the ability to interact with the messaging platforms. But one way or the other, it's usually a scheduled task that then communicates with the model using Camunda API.

Okay. Great. So seems to have a lot of interest behind it. What mechanisms are you using for the efficient history cleanup?

Okay. So we built the archival process. We used the design pattern and the [indiscernible] disruptor, I don't know if you've heard about it. It's quite powerful mechanism to control -- to implement and lock free concurrent processing. We are running on a single instance of the Spring Boot application. And that process is able to retrieve the data grouped by business key, as I mentioned before, generate the resulting document, publishing into document management and running in the loop on a fixed set of ring buffer elements, processing in parallel. But we're able to do it with a single process, and we are guiding 100,000 processes per day. So quite fast.

Wow. 100,000 a day?

Yes.

All right, not insignificant. Great. Gerry is asking, why are you querying the Camunda database directly? What's the reason behind that?

Very good question. Because sometimes Camunda API is not rich enough for us to do what we need to do. I'll give you examples where we have to construct a view for the users of the real-time data. And sometimes there is no API that would allow us to bring this data using just Camunda's native API that's provided to us. So in those cases, we'll have to use our own queries. One example I could give you is that when you need to bring in the comments into a task together with the task itself. So you could retrieve a set of tasks using the task service, but how do you add comments to your tasks? So you need to -- for every task, you would have to perform a separate query against the database when you need to get several thousand tasks from the user with the comments, you don't want to query the database several thousand times to bring comments every time. So you would want to perform direct join and get away with 1 or 2 queries instead of 1000s. So that would be one example. And there are plenty of use cases like this where you cannot really rely on Camunda offers. You have to go beyond that.

Thank you for that. Is the lack of database bottleneck in Camunda 8 a primary motivator free to migrate? What else is kind of part of that interest?

That's a heavy question. Now I have to start marketing Camunda 8. Okay. I wouldn't say that it's a primary motivator. It's one of the motivators. I would say the primary motivator is its distributed nature because no matter how much you try to scale Camunda 7 and before 8 Camunda versions, you would still be facing significant constraints associated with the single model where all your processor stored because at the end of the day, BPMN is a state machine. State has to [indiscernible] somewhere. Until Camunda 8, that state is persistent in a single database. So that is definitely a constraining factor. But there are other reasons that I would definitely consider migrating. I'm sure you heard the story half an hour ago about connectors and the power of bringing Camunda capabilities beyond just hard course of engineers. So those would be other drivers.

Okay. Thank you for that. Are you using Camunda's Tasklist? Or did you build something [indiscernible]? I think you actually might have touched on that a little bit...

We definitely built something [indiscernible]. So Camunda's Tasklist is pretty rudimentary and doesn't answer the needs of all clients. We've gone through several incarnations of user interfaces. Our users are quite sophisticated and perform very complex activities. And as a result, we could not use the Camunda's built-in capabilities from day 1. We knew that they will never fly. So -- I mean, our users are bringing sometimes 5,000, 10,000 tasks in front of them. So you [ can ] do it with Camunda's UI. [ Not ] possible.

Wow, that's remarkable. Okay. Are you using Optimize, Camunda Optimize, how or why not, I guess?

We're not. We started using activity and then migrated to Camunda. We stayed on the Community edition. We still are in a Community edition, even though we have the Enterprise relationship with Camunda. But when it comes to software, I would like running on Community edition because we just like, it's open source code. We can do anything we want with it. We're never constraining ourselves with the Enterprise editions. So that's probably one of the reasons why we never even looked at Optimize. We know about this product. We know all about its capabilities, but we never needed to put something fancy in order to determine where the bottlenecks are, we could query the database directly and find where those bottlenecks are without Optimize. And in fact, that's what Optimize is doing, querying the Camunda database and interpreting what it tells. So -- but it's a good product, I looked at it. It's just something that we've never used.