Slow Performance after importing years worth of transactions (edited)
Comments
-
+1 to system responsiveness/latecy…
I am new to Simplifi & comparing w/ Monarch Money (transitioning from Mint). Your system responsiveness/speed is very slow; specifically when manually updating/changing Category & Tags on individual Transactions. I am bringing this to your attention, because it a competitive disadvantage. Please "speed-up" your system/network responsiveness.0 -
Same issue with a very slow experience after all transaction data is imported.
1 -
I recently came across this thread too. I am astonished to see that these performance issues have been going on for two years without resolution. I just migrated from Mint and have only been using Simplifi for about a week. I followed Quicken's instructions for exporting and then importing my Mint data (16,000 transactions dating back to 2016). The instructions did not mention any issues with handling a high number of transactions.
Performance is slow on my Mac using Chrome, but my primary device is an iPad. I first downloaded the mobile app, but a popup message recommended using a browser because the app is not optimized for iPad. However, Simplifi is unusable on an iPad browser; I have tried both Chrome or Safari. On the iPad, even the most basic action like scrolling down the list of transactions in one account, causes Simplifi to hang and then crash. This occurs in both Safari and Chrome. Crash screenshots are attached.
Simplifi is going to great lengths to appeal to Mint users in need of a solution, and urgently need to address the product's ability to meet the needs of this new customer base.
0 -
Similar story to the latest comments. Transferred my data (~14K transactions) from mint and web application is way too slow - changing categories and payees takes a substantial amount longer than what it should likely take.
Troubling to say the least that Quicken Simplifi has been working on this for three years. I wanted to stick with Simplifi, but if issues persist it'd be difficult to stay.
0 -
It's amazing that there are 3 Quickens, Simplifi and Mint, [removed - inaccurate] and they can't share their software algorithms and programmers in India. With 5 versions of accounting programs they still can't transfer any categories and run imported program data from Mint. Just amazing, and sad.
0 -
@Coach Natalie
Are we still using the feedback method to submit our issue?
0 -
I have data from 2008 but no problem. Mine is iOS app. Not sure what going on.
Is this iOS app issue or android app or both?
0 -
Wow this had been an issue for almost 2 years and still no resolution. I just imported all my Mint transactions and it's painfully slow to do anything now.
0 -
Same here. Moving here after coming over from Mint - 15 years of transactions. An important part of my flow is categorizing transactions weekly at my desktop computer using the Web UI. The UI seems unresponsive during server fetches, which consistently are measured to take 500-1000ms. This would be a deal breaker for me to switch completely to Simplifi and I will be evaluating other competitors as well before Mint's shutdown. Any progress made by the product team will be much appreciated.
0 -
Hi. I had added a few comments to this thread (this morning) but they appear to be missing now.
I just wanted to point out that this issue is not related to importing large amounts of manual transactions. I just exported all of my Simplifi transactions, and I only have data going back to November 2019 (so it has only been 4 years) and I only have a total of about 9000 transactions over that 4 year period.
I think grouping this issue under the topic of "Slow Performance after importing years worth of transactions" isn't accurate since it seems like I and many others on this thread can reproduce the issue without doing manual imports at all.
1 -
I did some network captures in MS Edge and while I'm not an expert, one thing did stand out to me:
It looks like when marking a transaction as reviewed, "calc" makes a request to https://services.quicken.com/spending-watchlist/calc and that request in this case took 1.32s waiting for the server to respond. Editing the memo field generated the same request to the above url and took 1.57s. So, it looks like there is around a 1.5s minimum delay with any server interaction due to a call to this calc function and URL.
I captured the packet sequence to/from that ip address on a Notes field change, and got this:
That's 6 seconds from end to end on that server request/response. It looks like there are 2 large gaps between two sets of packets in the sequence, but since the traffic is encrypted there's not much sense I can make of it.
Interestingly… that call to calc happens after the PUT request where the new value for the Notes field is uploaded to the server. And that PUT only takes about 137ms.Quick update… I removed all of my Spending Watchlists (I only had one for Dining and Drinks anyhow), and after doing this, the response time for calling that calc URL is down to 100ms or so, instead of 1.5s. So, there is definitely something going on where every update to a transaction is calling calc and if you have any Spending Watchlists defined, it takes more than a second for calc to return.
Now that said, the UI was still slow, so fixing this didn't resolve the issue. A new packet capture still shows the overall request/response to be 6 or so seconds.
And yeah, a profile capture during a Notes update shows two long running tasks (each taking just under 3s) which matches the delays seen in the network packet capture:
Looking at some of the Event Log traces it seems like most of the time in those 3s blocks are spent in npm.crypto-js in string manipulation functions… either doing Base64 Encoding or actual encryption. Hmm…
Anyhow, hopefully some of this data is useful for debugging here. If there's any other debugging info that you think would be of assistance @Coach Natalie please let me know.
Cheers,
Perry
2 -
Another interesting find from the MS Edge DevTools…
This is in the Console when I try to do actions (edit notes, etc) in Simplifi. I asked Bard what "TTI for txnSave is too slow" is and got this answer:
"The Time to Interact (TTI) is a performance metric that measures the time it takes for a user to interact with a page after a user input event. The TTI for txnSave is too slow, which means that it is taking too long for the page to respond to the user's input. This can be frustrating for users, and it can also lead to a decrease in user satisfaction."
1 -
I think I figured it out by tracing through the javascript and here's what I found:
That might be hard to see but here's what it shows…
When you do ANY action in the UI, it eventually calls the encrypt function you see here highlighted with the red breakpoint. I looked at the values being passed to encrypt, which you can see on the far right under this: Object in the r: parameter.
I copied the value of r to a text file, and it was 17.1 MB large. And looking at the data, it appears to be ALL of my transactions across all history.
This means that for EVERY action in the UI, Simplifi is trying to re-encrypt the entire set of data. In short… if you have to edit something, you need to edit that entire blob of data, and then re-encrypt it. This seems like a really inefficient way to handle this for anything other than trivially sized datasets.
It also explains why people are mostly seeing this when they import large amounts of data manually.
To me this looks like a software design issue. The app needs to be rearchitected so that it doesn't require the entire dataset of transactions to be reencrypted with each action taken.
@Coach Natalie Thoughts?
Perry
2 -
I'd question: If data is being transmitted over an ssl type connection (the connection is encrypted) then does the data really need additional encryption on top of the ssl encryption. It seems like unnecessarily duplicating effort.
—
Rob Wilkens0 -
This data isn't even being sent over the wire as far as I can tell. When I did the tcp captures using Wireshark, I was only seeing a few hundred bytes transmitted during these operations. (See my Wireshark image posted in above comments) There was nothing even close to the size of the file being encrypted being sent over the wire.
I think this file is being re-encrypted entirely for local use? Not sure…
1 -
Sorry, I don't often use web browser debugging tools, but can you edit the live code and run it? i.e. if you changed the encrypt and decrypt functions there to do-nothing functions (return back what was passed in) do things noticeably improve?
I get the feeling that some young engineer thought they would protect their data structures from prying eyes by encrypting it, but doing it in code visible to the user in a debugger (which anyone trying to "decrypt" the code would have access to) as you demonstrated it shows what's being encrypted anyway.
—
Rob Wilkens0 -
@Coach Natalie This, to me, sounds like a bug. (Repeatedly) Encrypting and Decrypting things (large amounts of things) that just stay in the web browser's memory (never to be transmitted), and are otherwise accessible to someone with a browser debugging tool (hence, you're not hiding it from anyone), seems like a performance bug with potentially an easy fix (just don't encrypt/decrypt this structure). I will say that @perrynmeyers (my @ isn't working here) provided quite a service to you all by looking into this. I just hope it doesn't go without getting a fix (assuming they're right).
—
Rob Wilkens1 -
@perrynmyers you'll notice that @kai4387 came to a similar conclusion about local data encryption taking a significant portion of time back in April 2022 on page 1 of this post's comment thread. Seems like the same thing is still happening a year and a half later and is still a major bottleneck.
2 -
Thanks for mentioning that @josh_shannon, I had not seen @kai4387 comment on page 1. No idea how I missed it, as I thought I had read through the entire thread comprehensively. But you are correct, what @kai4387 suggested a year and a half ago is exactly what my debugging exercise showed as well. Credit to @kai4387 for figuring it out much sooner than I did. 😀
I'll note that I don't think I saw anyone from Simplifi reply to their comment, which included suggestions on how to improve the performance bottleneck with some simple changes. It's unclear to me why those changes were not investigated and implemented already.
I have no idea if live editing of the code in the debugger is possible. This was literally my first time ever using this tool in Edge/Chrome for anything serious. I had only ever used it before to capture things like OAUTH tokens from Google logins for accessing their APIs. 😀 But I'll take a look and see if that is possible.
2 -
Ok, this is really interesting.
I decided to redo the performance profile on both Edge and Chrome but after clearing my browser caches. Things were much more responsive, and when I tried profiling UI transactions/updates, I no longer saw any calls at all to encrypt. The breakpoints I had in the debugger were never being triggered.
Then I simply F5 refreshed the browser, and things immediately slowed down to where they were before. And now when I profile I see the calls to encrypt again.
What this tells me is that when someone logs into Simplifi on a clean browser (including an Incognito session) their performance will be good, because for some reason on that initial login it doesn't try to persist anything to a local cache.
But, once you close the window and reopen it (or just refresh the page), it reverts to the behavior where the local cache has to be updated and encrypted with every UI interaction.
I have absolutely no explanation for why this would be…
Actually, quick update… I can reproduce this without even doing a browser cache clear. Here's the steps:
- Log out of Simplifi and log back in. In this initial login session, the UI is responsive and profiling of UI interactions shows no encrypt calls.
- Refresh the browser and now UI interactions will start causing encrypt to be called, which slows down the UI responsiveness.
- Log out of Simplifi and log back in, and it'll go back to good UI responsiveness and no calls to encrypt on UI transactions.
Using an Incognito window of course simulates the above, since in a new Incognito window you're always logging in fresh. But the issue seems to be between the initial session created in the app on first login, vs. what happens when you close the window and reopen it (and therefore don't need to login again).
I just logged in fresh and have a good working instance right now (i.e. it is responsive and not calling encrypt). I'm going to let this window sit open overnight and see if it has the same behavior in the morning.
2 -
I love the many details provided by other posters on this thread. They provide ample explanations for what causes this latency. They basically did the investigation legwork for the SimplifyMoney development team in.
The least that Quicken could do in honor of the community's efforts around this ongoing issue is to make it a higher priority than it appears to have been (this thread was first created almost 3 years ago!).
This type of performance latency is not acceptable here in the 3rd decade of the 21st century. C'mon Quicken team, can we please put this topic to rest to the benefit of ALL users?
1 -
@perrynmyers @josh_shannon Another cause for latency (at least for me behind a proxy), is the absurd amount of calls to tracking / UX sites such as
clientstream.launchdarkly.com/…andsessions.bugsnag.com/. These tracking sites are called for every single character that I type in the transaction search bar and cause serious delay between typing and results. You can see in your URL bar that the things you type are appended to the query string as?displayNode=all&q=foobar(emphasis on thefoobar) and for whatever reason, the site is deciding to call out to these sites for every single letter. For me, the browser is waiting on the response from thissessions.bugsnag.com/between 150–250 ms PER LETTER.Or another example is when I cleared the search terms, Simplifi called out to
<LONG_STRING>.cloudfront.net/track/?verbose=1&ip=1&_=<LONG_STRING>with a whole host of tracking data (somewhat redacted below since I don't know what is personally identifying):[
{"event": "active","properties": {"$os": "Windows","$browser": "Microsoft Edge","$current_url": "https://app.simplifimoney.com/transactions?displayNode=all","$browser_version": 119,"$screen_height": 1080,"$screen_width": 1920,"mp_lib": "web","$lib_version": "2.47.0","$insert_id": "<STRING>","time": <FLOAT>,"distinct_id": "<STRING>","$device_id": "<STRING>","$initial_referrer": "$direct","$initial_referring_domain": "$direct","client_type": "Simplifi Web App","quicken_year": 2023,"$app_version_string": "3.92.0","$app_build_number": 22825,"$device": "desktop","$user_id": "<STRING>","qcs_id": "<STRING>","dataset_id": "<STRING>","dataset_platform": "ACME","dataset_created_by": "acme_ios","scope": "openid qcs.me qcs_acme_beta uaa.user","groups": "oauth.approvals,openid,password.write,qcs.me,qcs_acme_beta,scim.me,uaa.user","source": "QUICKEN","subscription_status": "subscribed","purchase_channel": "direct","subscription_tier": "acme-basic","subscription_tier_name": "Simplifi - Basic","subscription_state": "trial","subscription_term": "annual","auto_renew": "on","entitlements": [
"ACME_BASIC"
],"persist": true,"keep_logged_in": false,"proxy": true,"pathname": "/transactions","memory_device": 8589934592,"memory_limit": 4294705152,"memory_used": 103648109,"memory_available": 4191057043,"is_chromebook_twa": false,"token": "<STRING>","mp_sent_by_lib_version": "2.47.0"}}
]The above doesn't seem like much, but it's a 1,528 byte transaction which took 1.71s of queing, 286ms of connecting to cloudfront, and a further 115ms of waiting for cloudfront (a total of over 2.1s) just to clear my search bar!
There's also a site called
pendo.iothat I don't fully understand (they call themselves "The all-in-one platform for breakthrough digital experiences" enabling the dev team to "See the behavior behind the data. Watch video playbacks of how users navigate your app, so you can find the irrefutable evidence to make smarter, more impactful product decisions."), but is clearly another resource hogging tracking provider that's used by Simplifi, and it gets called a ton by the web app.Overall, the site just seems to be quite poorly designed with an insane amount of blocking, heavily invasive and tracking JS interactions and excessive overhead as you've found for the encryption for the entire transaction history. If you're going to be extremely invasive in your tracking, at least be fast like Google.
0 -
@molehill6804 But related to this thread, those 'tracking/ux' calls shouldn't the alter delay by the amount of data, so it might not be relevant to this. Not saying it's not causing a slowdown, just not related to importing years' worth of data.
—
Rob Wilkens1 -
Fair point @RobWilk … might be better suited to a new thread.
0 -
Hello All,
For anyone experiencing latency or lagging issues when using Quicken Simplifi, please continue to follow along here for updates. Our product team is aware of the issue and they are working to resolve it as quickly as possible. It sounds like they have a handle on things, but we'll be sure to reach out here if they request any additional data points.
We appreciate everyone's patience as we work to resolve this matter!
-Coach Natalie
1 -
@Coach Natalie Thanks!
Can confirm that engineering knows that the lag issues are definitely related to the encryption of the entire transaction store on every UI transaction?
And did you share with them the profiling data that I generated?
With this root cause analysis done, I think the issue should be fixable in a few days at the most. It just comes down to the developers deciding on an optimization path. For example, given my comment above, you could treat every "refresh" of the app the same as the initial login, meaning that there is no local caching of the transactions file needed.
Thanks!
Perry
0 -
Happens to me too. Android app and when I use the web browser on Chrome or Edge browser.
You guys clearly have latency issues. Your engineers should implement metrics to track round trip time between request and response. This is pretty bad.
For example when I try to edit a transaction, I have to wait 5-10 seconds before I can type in new values into the fields.
0 -
@esender See the analysis above. This isn't an issue with round trip networking latency, or even server delay. The issue in a nutshell is that on every client side interaction the Javascript in the browser is reencrypting the entire transaction store and persisting that to disk.
We've provided tracing to prove where the issue is (see above) and even suggested some ways they could design a solution to avoid the issue.
So far the only feedback we have gotten is "Our product team is aware of the issue and they are working to resolve it as quickly as possible.", and that was now a week ago.
0 -
Hey everyone!
Our product team is currently working on a new feature called "Transactions on Demand". This should clear up the processing that takes place on every single transaction and essentially "hide" older data from the processes, but still allow you to access that older data when needed. This should speed things up for everyone, and we can currently expect to see it in the next week or two. More to come — keep an eye on Updates From the Product Team for an announcement!
-Coach Natalie
4 -
This is awesome news @Coach Natalie ! Thanks for the update!
1
