Course migration progress

neoncube

7shi Hm, interesting…

If I try to download more than about 5 courses at a time (e.g. just downloading the HTML of https://app.memrise.com/community/course/<id>/course-name/), I start to get this response:

<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>

I'd assumed I was overloading the Memrise servers and backed back down to 5.

It looks like the Memrise audio and image files are stored on Amazon S3, though, and I can download about 50 of those files in parallel before getting 502 errors 🙂

I wonder if I'm being rate limited because of the large number of courses and S3 files that I've downloaded over the past few weeks.

Either way, I'm happy just downloading 5 courses in parallel for a week or so. Hopefully that'll be lighter on the Memrise servers, too 🙂

7shi

neoncube 502 occurs rather frequently. If I wait a little and try again, it almost always succeeds. I have included a countermeasure code for this.

async function fetchRetry(url, options, retries = 3, interval = 1000) {
let ret;
for (let i = 0; i < retries; i++) {
if (i) console.log("retry", i);
await sleep(i ? interval : 200);
try {
ret = await fetch(url, options);
if (ret.status != 502) break;
} catch (e) {
if (i == retries - 1) throw e;
}
}
return ret;
}

neoncube

7shi Thanks 🙂

Let me take another look at my code. When I'm downloading 5 courses in parallel, I get a 502 once in every, say, 10,000 courses, but if I download 6-7 courses in parallel, I get a 502 perhaps 30% of the time. Let me check if NodeJS supports persistent connections 🙂 Perhaps the issue is that I'm constantly closing and reopening connections.

neoncube

NewLandRise I was able to discover around 220,000 courses from the public courses categories 🙂

I'd totally forgotten that Memrise lets one set a course to "Incomplete" or "Unlisted"! Indeed, I'm just scanning the courses from 1 to 7,000,000'ish, and this might explain why I'm hitting so many courses that weren't listed via the public course categories! 🙂

I just took a quick look at an unlisted course, and it doesn't look like it has anything to indicate that it's unlisted. I can think of a couple of ways that we might be able to determine whether a course was incomplete/unlisted or not present on the course categories pages because it wasn't popular enough to be in the first 666 pages of a category:

Check to see if the course belongs to a category that at least 666 pages. If not, then the course should be incomplete/unlisted.
Download each user's profile and compile a list of all courses taught by all users. If a course is on this list, then it must be public. I'm not sure if this would work for courses where the author has already deactivated their account, though.

7shi

neoncube

I'm not sure if this would work for courses where the author has already deactivated their account, though.

Deleted users are shown as "deactivated user" and cannot view Teaching.

For example:

https://app.memrise.com/community/course/300000/bible-topics/

(This course has one of the highest number of levels I have found. ID might be a number intentionally targeted.)

neoncube

7shi That's pretty awesome they intentionally waited for that number, haha XD

It looks like enabling keep-alive fixed my 502 issue! ^_^ Thanks very much for pointing me in the right direction 🙂 This should make the Memrise servers a lot happier, too, I think, since they won't have to keep accepting and closing connections 🙂

neoncube

The amount of spam courses is truly amazing. Literally thousands, if not tens of thousands! For example, https://app.memrise.com/community/course/818880/how-to-purchase-zyban-without-prescription/

DW7

May I make a suggestion - I am a little concerned that incomplete and unlisted courses may be of no value (and the author who has set it as private, might not want it copied).

Basically with so many courses I hope people will be able to find the best (or most popular) courses.

Perhaps you could make it clear or even allow a filter that excludes incomplete and private courses.

I went looking for mine and without a function to search by a key word or author or text in the description field (I add "looked after by DW7" to make it easy for me and others to find courses and know that support is available) it will be hard for me to find them.

DW7

At present many are still missing.

This » one « should be under Literature.

And these » ones « should be under "Religion - Christianity".

Hi @[deleted], could you let me know if you have been able to upload these courses - I still can't find them (along with several Art courses I look after).

I noticed that a lot of the courses I created were not showing with the MemRise search by DW7 so I have added "Produced by DW7" to those courses, so a fresh » search « should list them all - some 275.

This one is also missing 7 Wonders ~ Ancient, Modern & Nature - by simone.zanotti - Memrise (under "Places" - not to be confused with the *100 version)

neoncube

At the request of some people on Reddit, I've added most Kurdish courses and some Japanese courses 🙂

neoncube

@DW7 I don't think those courses have been uploaded yet, but once they are, they should automatically show up in your dashboard, so you won't need to search for them 🙂 However, adding a search feature is definitely still on the to-do list! 🙂

Ya, once I found out that most of the courses that weren't listed in the categories were intentionally unlisted, I stopped planning to automatically add those courses to My Little Word Land, haha. God willing, I'm probably just going to add just the unlisted courses created by Memrise, since people have been requesting those, and then to keep copies of the other courses somewhere, in case someone asks for them in the future 🙂

7shi

neoncube This user seems to be a Memrise staff.

https://app.memrise.com/user/Decks/courses/teaching/

This user seems to manage some of the unlisted official courses. For example

https://app.memrise.com/community/course/1325838/arabic-2/

(Maybe there are other users like this.)

neoncube

Thanks, 7shi! Let me keep on eye on the Decks user, in particular 🙂

DW7

Good morning @[deleted],

once they are, they should automatically show up in your dashboard,

That's a new feature - brilliant - fantastic idea.

Incidentally I looked at the *100 course and you offer three options for testing whereas I had four (ie two plus reversed).

Location vs Name of image (eg Great Wall)
Name of image vs Photo

Then reversed.

Also possible Name vs Location and reversed.

(I have an image but the image button says Link! - PS but drag and drop worked.

Image description

neoncube

DW7 I think this is because I haven't yet added support for use images as an answer. Once I do, that testing mode should automatically be available 🙂

neoncube

Just finished uploaded courses for European Portuguese and Persian (Farsi), at the request of two Reddit users 🙂

neoncube

I've improved the upload script and uploaded another 2,627 non-language courses 🙂

@[deleted] This means that most (but perhaps not all) of your Christianity-related courses should now be available! 🙂 I'm not sure I'd go searching for them, though; I should be able to automatically assign them to your account, although it might take me a day or two 🙂

DW7

Thanks @[deleted] - I can see loads of the Bible courses I've created but they aren't on my Dashboard.

I have looked at a few and I don't have editing rights to them.

Searching for DW7 says there are none - although that is in the description.

Do you have enough information to assign them or do you have a way for searching for DW7 in the description?

Please also remember the

"This » one « should be under Literature".

DW7

I've also looked at Birds (under animals) but my two courses are not there.

My two butterfly courses are unloaded - thanks, and I have editing rights, but they are not on my Dashboard (automatically) - but if I start them, then they should be.

Still some art courses are still missing.

neoncube

@DW7 I have enough information to assign them, thanks 🙂 It may take me 1-2 days before I get to this, though.

« Previous Page Next Page »