**Notice**: Before we start I just want to say that this article is purely for educational purposes, please use it responsibly at your own risk.

`1337x.to` is a popular torrents website, with multiple categories: movies, music, and so on.

The goal of this article is to automate downloading torrents on a page like trending music, top-100 movies, and so on.

To download a torrent from `1337x.to` we need to go to the torrent details page, then click the magnet link, which will then be opened with your default torrent downloader.

Imagine you want to download 50 torrents on a page, you will have to visit each page separately, then download the files by clicking on the magnet link on that page.

I will focus on explaining the principles used to scrape the links from the `1337x.to`, so you will be able to reuse the same technologies and adapt them to your use case.

In this article, we will use Node.js with TypeScript support, even though it’s a small project I like to use TypeScript just for fun.

Get your terminal ready, and change your current directory to the location you want to store your projects.

We will create a new directory with `mkdir javascript_extract_torrents_magnet_links`, then change to it with `cd javascript_extract_torrents_magnet_links`.

Let's initialize our git repository `git init`, our npm repository `npm init -y`, and our tsconfig.json by installing TypeScript `npm install typescript --save-dev` then `npx tsc --init`. 

This is the `tsconfig.json` file after removing comments, and adding the `"outDir": ".out"` property.

When using Node.js with TypeSctipt it's helpful to install `@types/node` with `npm install @types/node@14 --save-dev` please specify your own version instead of 14.

Let's start by creating a new file at `src/app.ts` with a simple async function that logs hello world to the console.

The app function will be the main starting point of our application.

async function app() {
  console.log("hello world");
}

app();

To execute it, we will add an npm script called `fetch` to compile and execute our app.ts file.

{
    ...
    "scripts": {
      "fetch": "tsc && node .out/app.js"
    },
    ...
}

Because we use git, we need to add a `.gitignore`

To test that everything is working as expected we will run `npm run fetch` we should see a *hello world* message.

It's time to save our change to our git repository with `git add .` then `git commit -m "initial commit"`.

Web scraping links for torrent detail pages

As we said before, there are two types of pages we will scrape, a listing page that contains multiple links to detail pages, and a single detail page.

Let's first install both dependencies with `npm install cheerio node-libcurl` and `npm install @types/cheerio --save-dev`.

The main idea behind web scraping is fetching the HTML content for a page, then extracting the required information in an automatic way.

The same principle applies here, this is the structure of links on the "/popular-music" page.

<table class="...">
  <thead>
    ...
  </thead>
  <tbody>
    <tr>
      <td class="...">
        <a href="..." class="icon"><i class="flaticon-hd"></i></a>
        <a href="the-link-we-are-looking-for">Torrent title</a>
        <span class="..."><i class="..."></i>1</span>
      </td>
      ...
    </tr>
  </tbody>
</table>

To extract the links we're looking for, we will need a css selector to use with `cheerio` to query the exact anchor elements of our choice.

To make it readable and clean we will create a new object `selectors` which will store all of our css selectors, this becomes handy in larger web scraping projects.

It's always easier to explain after the code is already written, let's do just that and then explain what's going on.

import * as cheerio from "cheerio";
import { curly } from "node-libcurl";

const config = {
  baseUrl: "https://1337x.to",
  selectors: {
    anchorLink:
      'table > tbody> tr > td:first-child > a:last-child[href^="/torrent/"]',
    magnetAnchorLink: 'ul>li>a[onclick="javascript: count(this);"]',
  },
};

async function get(path: string): Promise<string> {
  const result = await curly.get(`${config.baseUrl}${path}`);
  if (result.statusCode !== 200) {
    throw `error with status code: ${result.statusCode}`;
  }
  return result.data;
}

....

Because this project is not that big we will keep all code in a single file, but please feel free to move any helper function to a `helpers.ts` file, and wrappers for packages like our `get` function to `lib.ts`.

The important thing is to be comfortable with your file structure, so it's easy for you to modify or add new features.

As you can see we have declared a new object `config` which stores our app configurations, we have a `baseUrl` which is the homepage of the website we are scraping.

And the `selectors` config which includes the css selectors we need to extract data from the scrapped HTML content.

Then we created a wrapper function around our `curly` function from the `node-libcurl` npm package.

This way it's easy to use and without unnecessary duplication, it will also give us the possibility to easily replace the current package with another one without having to do changes all over the place.

The `get` function will simply try to fetch the HTML content from the page path, by concatenating it with the `baseUrl`, it will return the HTML in case of success otherwise it will just throw an error.

...

async function app(path: string) {
  const html = await get(path);
  const $ = cheerio.load(html);

  const anchors = Array.from($(config.selectors.anchorLink));
  const hrefs = anchors.map((a) => $(a).attr("href")) as string[];

  console.log({ hrefs });
}

app("/popular-music");

Inside our main `app` function, we accept the `path` of the page we want to scrape, we use the `get` function to fetch the page HTML content.

Then we will create a dom instance using `cheerio.load` passing to it the HTML content we got back from `curl`. `cheerio.load` returns something similar to `jQuery` but on the server.

Now it's time to extract some data, using our `anchorLink` selector we will select all the anchor elements related to the torrent detail pages.

This will return something called `cheerio.Cheerio` which contains the elements that match our selector, used with `Array.from` we get an array of `cheerio.Element[]`.

You can imagine the `cheerio.Element` as an `HTMLAnchorElement` and object with the properties we need like the `href` property in our case.

Using the `Array.prototype.map() ` we map the `cheerio.Element[]` to elements `href` attribute `string[]`.

At this point, if you run the `fetch` script with `npm run fetch`, you should see an array of the href attribute values logged to your console.

Web scraping magnet link from each torrent detail page

Now that we have the list of relative links to all torrent detail pages, we can scrape each page and extract the torrent magnet link needed to download the torrent files.

Just after where we map elements to links, we will add the following.

  ...
  const hrefs = anchors.map((a) => $(a).attr("href"));

  let fetched = 1;
  const magnets: string[] = [];
  for (const href of hrefs) {
    const html = await get(href);
    const $ = cheerio.load(html);

    const magnet = $(config.selectors.magnetAnchorLink).attr("href") as string;
    magnets.push(magnet);

    console.log("magnet fetched: ", fetched);
    fetched++;
  }

  console.log({ magnets });
  ...

We loop through the `hrefs` array from the previous step, and for each `href` we do a similar thing as we did before.

We fetch the page HTML content using the `get` function, then we create a cheerio instance with the `cheerio.load` function.

Here because we only have a single link per page, we don't need to map anything, we just access the `href` attribute and push it to the array `magnets`.

We also declare the `fetched` variable which starts at `1` and increments by `1` each time a page was scrapped. We then `console.log` it to have some feedback.

At this point, if you run `npm run fetch` again, you should see an array of the magnet links we extracted from each torrent detail page.

Create an HTML page with our magnet links

The purpose of doing all of this was to be able to automate downloading torrents through extracting magnet links and using them in an automatic fashion.

In this section, we will have our app generate a simple HTML page with an anchor link to each magnet link.

The idea is to create a `public/index.html` file, populate it with the links we extracted, then server using the `live-server` package.

When this page loads we will have it click on the links every 3 seconds, at first you will need to allow the page to open links (you will get a notice that links were blocked, so you need to allow all).

By default when you click a magnet link on your browser you get a popup asking you if you want to open it with your default torrent downloader or cancel.

What you want to do is check the *Always allow site_domain to open links ...* and click *Open torrent_app_name* this will allow our script to work as we intended to use it.

Enough theory let's write some code, we will use `fs/promises` to create and put content to our file by creating a new `writePublicIndex` function that takes the scrapped magnet links.

Just after the `get` function add the following.

// imports
import path from 'path'
import fs from 'fs/promises'

...

async function writePublicIndex(magnets: string[]) {
  const indexPath = path.resolve("./public/index.html");

  const html = ['<div>'];
  for (const magnet of magnets) {
    html.push(`
  <a href="${magnet}" target="_blank">Link</a>`);
  }
  html.push(`
  <script>
    window.onload = async (event) => {
      let executed = 1
      for (let anchor of Array.from(document.getElementsByTagName('a'))) {
        await new Promise((res, rej) => {
          setTimeout(() => {
            anchor.click()
            console.log('magnet executed: ', executed)
            res()
          }, 3000)
        })
        executed++
      }
    }
  </script>
</div>
  `);

  try {
    await fs.writeFile(indexPath, html.join(""), {
      encoding: "utf-8",
    });
  } catch (e: any) {
    console.log(`Error while writing the ${indexPath}: ${e.message}`);
  }
}

...

For the `writePublicIndex` function to work properly we need to manually create the `public/index.html` file and leave it empty.

At first, we declare and set the `indexPath` constant with the help of the `path.resolve` function, which enables us to get an absolute path from a relative one.

Then we declare an `html` array, which we use to construct our HTML page, first we push a `<div>` to it, then we use the array `map` prototype to map our magnet links to HTML anchor tags.

After that, we add a script that will run after the page has loaded, the script's purpose is to click on each link on the page with a delay of 3 seconds.

After that, we just close our `</div>` tag, and we try to write the content to the `public/index.html` file, if an error occurs like the *file does not exist*, we just log the error message to our terminal.

After creating the `writePublicIndex` function, it's time to make use of in order to populate our `public/index.html` with the magnet links we scrapped in the previous section.

Locate where we logged the magnets array to the console like so `console.log({ magnets });` change it with `await writePublicIndex(magnets);`.

...

async function writePublicIndex(magnets: string[]) {
  ...
}

async function app(path: string) {
  ...

  for (const href of hrefs) {
    ...
  }

  await writePublicIndex(magnets);
}

app("/cat/Games/1/");

Now, if we run our `fetch` script again it should populate our `public/index.html` file with links to all the magnet links we scrapped.

This automation wouldn’t be complete without making the page path dynamic, in our example we use a static path like `"/cat/Games/1/"` or `"/popular-music"`.

But what about passing it as an argument from the terminal, and maybe having a default page path in case no page was provided.

It's pretty straightforward, we will use the `process.argv` object to extract the first argument we provide to our app.

At the end of the `src/app.ts` file, let's change this `app("/cat/Games/1/");` with the following.

...

const pagePath = process.argv.length >= 3 ? process.argv[2] : "/popular-music";
app(pagePath);

The first argument we pass, the third in the `process.argv` as the two args are reserved for the node binary path, and the script file path.

So we check if we have more than 3 args, if so we take the third argument otherwise we use the `"/popular-music"` as the default page path.

Let's first install it as a dev dependency using `npm install live-server --save-dev` this package will help us quickly serve our public directory which will serve by default the `index.html` file.

Cause we're using typescript, we also need to install its types by `npm i --save-dev @types/live-server`.

So we will have a single npm script that does everything for us, from fetching and scraping the magnet links to serving them and opening them with our default torrent downloader.

After the installation is complete, we will create a new function `servePublicDir` which will do what we explained above. 

Just before our `app` function, we will define our function as shown below.

// imports
import liveServer from "live-server";

...

async function servePublicDir() {
  const params = {
    root: "public",
    open: false,
  };
  liveServer.start(params);
}

Nothing much to explain here, its self-explanatory we use the `start` method on the `liveServer` object we imported.

Now we just need to use it, after the `writePublicIndex` completed writing to our `public/index.html` file.

I also added some exception handling for our application, so we won't serve the page if the scraping wasn’t successful.

This is the `app` function content, after all the modifications we did.

...

async function app(path: string) {
  try {
    const html = await get(path);
    const $ = cheerio.load(html);

    const anchors = Array.from($(config.selectors.anchorLink));
    const hrefs = anchors.map((a) => $(a).attr("href")) as string[];

    let fetched = 1;
    const magnets: string[] = [];
    for (const href of hrefs) {
      const html = await get(href);
      const $ = cheerio.load(html);

      const magnet = $(config.selectors.magnetAnchorLink).attr(
        "href"
      ) as string;
      magnets.push(magnet);

      console.log("magnet fetched: ", fetched);
      fetched++;
    }

    await writePublicIndex(magnets);
    servePublicDir();
  } catch (e: any) {
    console.error(e);
  }
}

const pagePath = process.argv.length >= 3 ? process.argv[2] : "/popular-music";
app(pagePath);

Now you can use it with something like `npm run fetch /cat/Music/1/` and it should scrape all the torrent magnet links for the  **/cat/Music/1/** page.

A practical guide on how you can use the cheerio and curl npm packages to web scrape magnet links from a torrent website with Node.js and TypeScript.

<strong>Notice</strong>: Before we start I just want to say that this article is purely for educational purposes, please use it responsibly at your own risk.


<code>1337x.to</code> is a popular torrents website, with multiple categories: movies, music, and so on.


The goal of this article is to automate downloading torrents on a page like trending music, top-100 movies, and so on.


To download a torrent from <code>1337x.to</code> we need to go to the torrent details page, then click the magnet link, which will then be opened with your default torrent downloader.


Imagine you want to download 50 torrents on a page, you will have to visit each page separately, then download the files by clicking on the magnet link on that page.


I will focus on explaining the principles used to scrape the links from the <code>1337x.to</code>, so you will be able to reuse the same technologies and adapt them to your use case.


In this article, we will use Node.js with TypeScript support, even though it’s a small project I like to use TypeScript just for fun.


Get your terminal ready, and change your current directory to the location you want to store your projects.


We will create a new directory with <code>mkdir javascript_extract_torrents_magnet_links</code>, then change to it with <code>cd javascript_extract_torrents_magnet_links</code>.


Let's initialize our git repository <code>git init</code>, our npm repository <code>npm init -y</code>, and our tsconfig.json by installing TypeScript <code>npm install typescript --save-dev</code> then <code>npx tsc --init</code>.


This is the <code>tsconfig.json</code> file after removing comments, and adding the <code>"outDir": ".out"</code> property.


When using Node.js with TypeSctipt it's helpful to install <code>@types/node</code> with <code>npm install @types/node@14 --save-dev</code> please specify your own version instead of 14.


Let's start by creating a new file at <code>src/app.ts</code> with a simple async function that logs hello world to the console.


The app function will be the main starting point of our application.


To execute it, we will add an npm script called <code>fetch</code> to compile and execute our app.ts file.


Because we use git, we need to add a <code>.gitignore</code>


To test that everything is working as expected we will run <code>npm run fetch</code> we should see a <em>hello world</em> message.


It's time to save our change to our git repository with <code>git add .</code> then <code>git commit -m "initial commit"</code>.


As we said before, there are two types of pages we will scrape, a listing page that contains multiple links to detail pages, and a single detail page.


Let's first install both dependencies with <code>npm install cheerio node-libcurl</code> and <code>npm install @types/cheerio --save-dev</code>.


The main idea behind web scraping is fetching the HTML content for a page, then extracting the required information in an automatic way.


The same principle applies here, this is the structure of links on the "/popular-music" page.


To extract the links we're looking for, we will need a css selector to use with <code>cheerio</code> to query the exact anchor elements of our choice.


To make it readable and clean we will create a new object <code>selectors</code> which will store all of our css selectors, this becomes handy in larger web scraping projects.


It's always easier to explain after the code is already written, let's do just that and then explain what's going on.


Because this project is not that big we will keep all code in a single file, but please feel free to move any helper function to a <code>helpers.ts</code> file, and wrappers for packages like our <code>get</code> function to <code>lib.ts</code>.


The important thing is to be comfortable with your file structure, so it's easy for you to modify or add new features.


As you can see we have declared a new object <code>config</code> which stores our app configurations, we have a <code>baseUrl</code> which is the homepage of the website we are scraping.


And the <code>selectors</code> config which includes the css selectors we need to extract data from the scrapped HTML content.


Then we created a wrapper function around our <code>curly</code> function from the <code>node-libcurl</code> npm package.


This way it's easy to use and without unnecessary duplication, it will also give us the possibility to easily replace the current package with another one without having to do changes all over the place.


The <code>get</code> function will simply try to fetch the HTML content from the page path, by concatenating it with the <code>baseUrl</code>, it will return the HTML in case of success otherwise it will just throw an error.


Inside our main <code>app</code> function, we accept the <code>path</code> of the page we want to scrape, we use the <code>get</code> function to fetch the page HTML content.


Then we will create a dom instance using <code>cheerio.load</code> passing to it the HTML content we got back from <code>curl</code>. <code>cheerio.load</code> returns something similar to <code>jQuery</code> but on the server.


Now it's time to extract some data, using our <code>anchorLink</code> selector we will select all the anchor elements related to the torrent detail pages.


This will return something called <code>cheerio.Cheerio</code> which contains the elements that match our selector, used with <code>Array.from</code> we get an array of <code>cheerio.Element[]</code>.


You can imagine the <code>cheerio.Element</code> as an <code>HTMLAnchorElement</code> and object with the properties we need like the <code>href</code> property in our case.


Using the <code>Array.prototype.map() </code> we map the <code>cheerio.Element[]</code> to elements <code>href</code> attribute <code>string[]</code>.


At this point, if you run the <code>fetch</code> script with <code>npm run fetch</code>, you should see an array of the href attribute values logged to your console.


Now that we have the list of relative links to all torrent detail pages, we can scrape each page and extract the torrent magnet link needed to download the torrent files.


Just after where we map elements to links, we will add the following.


We loop through the <code>hrefs</code> array from the previous step, and for each <code>href</code> we do a similar thing as we did before.


We fetch the page HTML content using the <code>get</code> function, then we create a cheerio instance with the <code>cheerio.load</code> function.


Here because we only have a single link per page, we don't need to map anything, we just access the <code>href</code> attribute and push it to the array <code>magnets</code>.


We also declare the <code>fetched</code> variable which starts at <code>1</code> and increments by <code>1</code> each time a page was scrapped. We then <code>console.log</code> it to have some feedback.


At this point, if you run <code>npm run fetch</code> again, you should see an array of the magnet links we extracted from each torrent detail page.


The purpose of doing all of this was to be able to automate downloading torrents through extracting magnet links and using them in an automatic fashion.


In this section, we will have our app generate a simple HTML page with an anchor link to each magnet link.


The idea is to create a <code>public/index.html</code> file, populate it with the links we extracted, then server using the <code>live-server</code> package.


When this page loads we will have it click on the links every 3 seconds, at first you will need to allow the page to open links (you will get a notice that links were blocked, so you need to allow all).


By default when you click a magnet link on your browser you get a popup asking you if you want to open it with your default torrent downloader or cancel.


What you want to do is check the <em>Always allow site_domain to open links ...</em> and click <em>Open torrent_app_name</em> this will allow our script to work as we intended to use it.


Enough theory let's write some code, we will use <code>fs/promises</code> to create and put content to our file by creating a new <code>writePublicIndex</code> function that takes the scrapped magnet links.


Just after the <code>get</code> function add the following.


For the <code>writePublicIndex</code> function to work properly we need to manually create the <code>public/index.html</code> file and leave it empty.


At first, we declare and set the <code>indexPath</code> constant with the help of the <code>path.resolve</code> function, which enables us to get an absolute path from a relative one.


Then we declare an <code>html</code> array, which we use to construct our HTML page, first we push a <code>&lt;div&gt;</code> to it, then we use the array <code>map</code> prototype to map our magnet links to HTML anchor tags.


After that, we add a script that will run after the page has loaded, the script's purpose is to click on each link on the page with a delay of 3 seconds.


After that, we just close our <code>&lt;/div&gt;</code> tag, and we try to write the content to the <code>public/index.html</code> file, if an error occurs like the <em>file does not exist</em>, we just log the error message to our terminal.


After creating the <code>writePublicIndex</code> function, it's time to make use of in order to populate our <code>public/index.html</code> with the magnet links we scrapped in the previous section.


Locate where we logged the magnets array to the console like so <code>console.log({ magnets });</code> change it with <code>await writePublicIndex(magnets);</code>.


Now, if we run our <code>fetch</code> script again it should populate our <code>public/index.html</code> file with links to all the magnet links we scrapped.


This automation wouldn’t be complete without making the page path dynamic, in our example we use a static path like <code>"/cat/Games/1/"</code> or <code>"/popular-music"</code>.


But what about passing it as an argument from the terminal, and maybe having a default page path in case no page was provided.


It's pretty straightforward, we will use the <code>process.argv</code> object to extract the first argument we provide to our app.


At the end of the <code>src/app.ts</code> file, let's change this <code>app("/cat/Games/1/");</code> with the following.


The first argument we pass, the third in the <code>process.argv</code> as the two args are reserved for the node binary path, and the script file path.


So we check if we have more than 3 args, if so we take the third argument otherwise we use the <code>"/popular-music"</code> as the default page path.


Let's first install it as a dev dependency using <code>npm install live-server --save-dev</code> this package will help us quickly serve our public directory which will serve by default the <code>index.html</code> file.


Cause we're using typescript, we also need to install its types by <code>npm i --save-dev @types/live-server</code>.


So we will have a single npm script that does everything for us, from fetching and scraping the magnet links to serving them and opening them with our default torrent downloader.


After the installation is complete, we will create a new function <code>servePublicDir</code> which will do what we explained above.


Just before our <code>app</code> function, we will define our function as shown below.


Nothing much to explain here, its self-explanatory we use the <code>start</code> method on the <code>liveServer</code> object we imported.


Now we just need to use it, after the <code>writePublicIndex</code> completed writing to our <code>public/index.html</code> file.


I also added some exception handling for our application, so we won't serve the page if the scraping wasn’t successful.


This is the <code>app</code> function content, after all the modifications we did.


Now you can use it with something like <code>npm run fetch /cat/Music/1/</code> and it should scrape all the torrent magnet links for the  <strong>/cat/Music/1/</strong> page.


Explore JavaScript Articles

Learn JavaScript with practical articles

Web Scraping magnet links from a website with cheerio and curl

JavaScript