Review times (possible bug--not sure)

taubercat

Hi,

So I'm at the point where in https://mylittlewordland.com/course/454845/1-russian-random-beginner-intermediate-vocab I have words with "remembered" times greater than 7 days. I thought the time until the next review was the following: if I correctly remember a word for 2 days, for example, it will have me review at 2 x that time, so 4 days later. (Correct me if I'm wrong.)

However, after 7 days that seems to no longer be the case. Words that had been marked as "remembered for 7 days" are, after the next successful review, marked as "remembered for 10 days." Also, words that were marked as "remembered for 10 days" prior to successful review are marked as "remembered for 11 days" afterward. In other words, either what is displayed to me is incorrect (but I don't think that's the case, because I was genuinely surprised that the program was presenting me those words already for review) or it no longer doubles after 7 days (or I understood the algorithm incorrectly). I've been reviewing 2-3 times a day, so it's not because I was behind on reviews and managed to remember longer than the program expected me to. Thing is, if the program will have me review 7 days > 10 days > 11 days > ? days then the amount of words to review is just going to become impossible. I'll have several hundred words to review several times a day, and that doesn't feel efficient, nor is it doable…

Thanks!

neoncube

taubercat Thank you very much for letting me know about this! 🙂

It looks like I was incorrect: Words are suggested for review after 1.5x time has passed, rather than 2x.

Another problem is that the algorithm uses the time since a word was last reviewed, but the website only shows the total time remembered since learning the word. For example, here are the times you reviewed the word здорово and the associated data that My Little Word Land is tracking:

Reviewed at	Time since last review	Total time remembered
2025-01-10 15:57	0	Just learned
2025-01-10 17:31	2 hours and 34 minutes	2 hours and 34 minutes
2025-01-11 14:49	21 hours 18 minutes	22 hours 52 minutes
2025-01-13 09:32	1 day, 18 hours, and 43 minutes	2 days, 17 hours, 35 minutes
2025-01-16 08:29	2 days, 22 hours, 57 minutes	5 days, 16 hours, 32 minutes
2025-01-21 18:37	5 days, 10 hours, 8 minutes	11 days, 2 hours, 40 minutes

The algorithm marks a word as being ready for review when more time has passed than 1.5 x "Time since last review". For example, здорово would be up for review again when:

5 days 10 hours, 8 minutes x 1.5 has passed
5 days 10 hours 8 minutes x 1.5 = 8 days 3 hours 12 minutes
2025-01-21 18:37 + 8 days 3 hours 12 minutes = 2025-01-29 21:49

Therefore, здорово would be up for review again on 2025-01-29 21:49 . This would make the total time remembered be 19 days 5 hours 52 minutes.

I think there are a few issues:

The algorithm is using 1.5x instead of 2x. Let me see if I can quickly add a setting for adjusting this 🙂
The "Remembered" column that's displayed on a course's Overview page isn't what's used to calculate when a word is ready to be reviewed, which is confusing.
The "Remembered" column is always cut down to just one word. For example, "5 days, 16 hours, 32 minutes" is shown as just "5 days". I did this to keep the column narrow, but in some cases, cutting off the more detailed time information might be confusing.

Let me think about how to fix this. I think adding an option to adjust the review time multiplier should be a good start 🙂

taubercat

Thank you! This explains a lot. I see you've already changed the remembered column. That's very helpful! I think it's fine with the shortened form of the remembered column (5 days isntead of 5 days, 16 hours) since I'm looking at those numbers mostly for an idea of how well I've mastered the word, and a quick glance over 5 days, 7 days, 11 days is going to be more informative than if I have to wade through 5 days 7 hours 3 minutes, 7 days 4 hours 20 minutes, 11 days 13 hours 28 minutes, etc.

For the adjusting the review time multiplier--I really like that idea. Another idea, if you have time, is to have words that are causing trouble (which could be defined as having been answered incorrectly at least once in the last 2 times it was presented) be shown more frequently until they are no longer causing trouble (ie, when the word has been produced correctly at least the last two times it was asked). I have no idea how complicated that would be to set up. Maybe for that, too, it'd be possible to let the user decide the frequency, ie 1.5X for difficult words and 2X for the others?

neoncube

taubercat You're welcome! 🙂 Thanks again for prompting me to look into this!

Agreed about the "remembered" column. In fact, instead of text, that column used to show a picture of a rock that got larger the longer a word had been remembered. I still kind of like that idea 🙂

I've added an option on https://beta.mylittlewordland.com/account for adjusting the review time multiplier:

Image description

Note that this option currently only effects the beta site. If you set it to 2, for example, then words on the beta site will be shown if more than 2x time has elapsed, but on the normal site they'll still be shown when more than 1.5x time has elapsed.

I like the idea of having difficult words and possibly having a different frequency for them 🙂 Your algorithm for deciding when a words if registered as difficult seems good, too 🙂 I'll make a note of this.

taubercat

Thanks so much! The review time adjustment slider works great. I'm currently testing how I fare at 1.7x, but the nice thing is that everyone can now adjust it based on their memorization abilities. Thanks!

neoncube

taubercat You're welcome, and thanks again for helping me dig into this! 🙂

It'll be interesting to see what multipliers end up being popular 🙂

I think it'd be cool in the future to make this be applicable on a per-language basis, too, so that if people were learning different languages, they could have different multiplier for each one.

taubercat

Perhaps even by course. Not all courses are set up as the same difficulty--for example, I have one that's just plain vocab: 1 English word-->1 Russian word, which is pretty simple, and a second one that's about remembering Russian word pairs (so several words, in the right order, with stress marks), which is far more difficult and prone to errors, so reviewing that more frequently is going to be more effective. Thanks for all you do!

neoncube

taubercat Good point!

Ideally, I think that it'd be nice for essentially all settings to be configurable per language, per testing direction per course, and even per testing direction within a specific course. I've been thinking about some ways to make an interface for that which wouldn't be super confusing 🙂

taubercat

You'll probably need some excellent headers, ie:

Settings for all courses

Settings ONLY for this course (this overrides settings for all courses)

neoncube

taubercat Yes, indeed! 🙂 I'm thinking that it might also be useful to have an option to save the current settings to the course, language, or account level, too.

P.S. Sorry for the late reply XD

Bonzo

Hi @neoncube.

My post refers to your first answer in this thread (the one with the table).

I always wondered how the "ready to review" algo works because it didn't behave as I expected – more so after I had discovered the "Review time multiplier" setting and set it to 1.1 hoping to get more frequent reviews.

I find it very surprising to read that the "time until the next review is ready" is said configurable factor times "Time since last review".

A simple example to demonstrate what I mean, using a factor of 1.5: Let's assume I reviewed a word on Jan 1st and the review delay is 2 days, so the word becomes ready for review on Jan 3rd. In case I review it on Jan 3rd, it will be ready for review again on Jan 6th, which is a review delay of 3 days = 1.5 * 2 days. Perfect.

But what if I don't do the review on Jan 3rd (no time, illness, vacation, ...)? Let's assume I miss another 10 days and do the review on Jan 13th. Your algo will (if I understand it correctly) schedule the next review for Jan 31st, because the total time since the last review was 12 days (1st to 13th), 12*1.5=18 and 13th+18=31st.

At least that's my interpretation of you explanations of the algorithm. Please correct me if I got it wrong.

I'd rather suggest the following (different) behavior: The scheduled review delay of 2 days (1st to 3rd) is multiplied by the factor --> 2*1.5=3, so the new delay is 3 days, no matter when I actually do the review. So when I do the review on Jan 13th, the algo should schedule the next review on Jan 16th (=13th + 3).

A delay of 3 days represents my learning progress but 18 days doesn't.

What are your thoughts about this?

taubercat

@neoncube if you choose to follow Bonzo's suggestion, could you keep the current setup as an option in the settings? I really like that if after say 10 days I still remember a word that the program expected me to know for only 3, it will mark it as such and ask me again after 15 days or whatever I set the bar to. Memrise obligatorily made it be 3*whatever their multiplier was, which meant I'd end up with waves of huge amounts of words to review if I missed out on even just one day of reviewing. I always felt like I ended up reviewing stuff that I really didn't need to review and was losing a lot of time on that.