
Recently, I found myself in a situation where I needed to convert a big list of permanent Twitter user IDs to usernames. Getting the username from an ID is as easy as following what this URL redirects to: [https://twitter.com/intent/user?user_id=\<ID\>](https://twitter.com/intent/user?user_id=12). But doing this is bulk requires using the Twitter API in some way. Registering for Twitter developer account requires verifying your phone number and I didn't want to deal with credentials for this temporary task.

Meanwhile, I found [tweeterid.com](https://tweeterid.com). It allows you to convert Twitter IDs to screen names but you have to do it one by one.

Inspecting the network tab in the developer console, I found that it makes a POST request to `https://tweeterid.com/ajax.php` with the data `input=<ID>`. I made a file named `twids` with each line containing one ID, and ran the following command in my terminal. It takes each line of `twids` file, makes a POST request with the data, and writes out the response to stdout separated by newline characters.

```shell
xargs -I {} curl -s -d 'input={}' 'https://tweeterid.com/ajax.php' -w '\n' < twids
```

If you're wondering how I ended up with this problem, it all started when I decided to wipe out my Twitter profile and downloaded an archive of my data. I soon discovered that the file containing the list of accounts I followed only included the corresponding Twitter IDs, and I needed the usernames in order to obtain RSS feeds for Twitter accounts from [nitter.net](https://nitter.net).


Bulk convert Twitter IDs to usernames without Twitter API

[Top]


A while ago, I published [pl-sort](https://pl-sort.netlify.app), a web app that enables users to sort YouTube playlists based on various metrics such as views, likes/comments count, and more. Initially, creating this tool was not my goal. I simply wanted to sort a playlist. Here is the method I used, which involves a single bash oneliner using standard command line tools.

Suppose we want the videos in [this playlist](https://www.youtube.com/playlist?list=PLZHQObOWTQDMsr9K-rj53DwVRMYO3t5Yr 'The essence of calculus - 3Blue1Brown') sorted by their view counts in this format:

```json
[
  {
    "title": "The essence of calculus",
    "views": 7340027
  },
  {
    "title": "The other way to visualize derivatives | Chapter 12, Essence of calculus",
    "views": 3339209
  },
  ...
]
```

We can do this using tools that are readily available on most Unix-like systems:

- `curl`: A command line HTTP client used to send requests to endpoints.
- `jq`: A JSON processor that lets you transform JSON data into desired format with a powerful syntax. We will use jq to transform the response into the output format we want and sort the output by various keys. If your system doesn't have jq installed, see [jq's installation guide](https://stedolan.github.io/jq/download/).
- `xargs`: Commands like `grep` or `awk` can take input from command line arguments as well as `stdin` (output from one command can be piped into them as input) but `curl` can only take input as arguments. We will use xargs to take each video ID from the playlist, contruct an endpoint that we can send requests to, and pass it as an argument for curl.

Because of how YouTube's Data API v3 works, we cannot get all the information about every video in a playlist in a single request. First, we need to hit the [playlistItems](https://developers.google.com/youtube/v3/docs/playlistItems/list) endpoint to get the relevant video IDs, then call the [videos](https://developers.google.com/youtube/v3/docs/videos/list) endpoint for each ID for details.

Before you begin, obtain an API key from [Google Developer Console](https://console.cloud.google.com/) for YouTube Data API. Then set it as an environment variable by running the following command:

```bash
# Don't add any spaces around the = sign
$ export YT_KEY="your api key"

# Add these as well in order to shorten our command
$ export YT_API="https://www.googleapis.com/youtube/v3"
$ export YT_LISTID="PLZHQObOWTQDMsr9K-rj53DwVRMYO3t5Yr" # Playlist ID
```

By doing this, we don't have to type in these frequently needed values in our command every time we run it. We can refer to them with their keys when we need to, like this:

```bash
# -s flag makes curl silent
$ curl -s "$YT_API/playlistItems?part=contentDetails&playlistId=$YT_LISTID&maxResults=50&key=$YT_KEY"
{
  "kind": "youtube#playlistItemListResponse",
  "etag": "jhS4gzCu5OcX8oftAWInXrLHoZs",
  "items": [
    {
      "kind": "youtube#playlistItem",
      "etag": "ppImKV3lyokspBcbXkImZ7cmdEk",
      "id": "UExaSFFPYk9XVFFETXNyOUstcmo1M0R3VlJNWU8zdDVZci41NkI0NEY2RDEwNTU3Q0M2",
      "contentDetails": {
        "videoId": "WUvTyaaNkzM",
        "videoPublishedAt": "2017-04-28T15:58:48Z"
      }
    },
    {
      "kind": "youtube#playlistItem",
      "etag": "a29dMgyS-DwQYMokKGzUXy4H26U",
      "id": "UExaSFFPYk9XVFFETXNyOUstcmo1M0R3VlJNWU8zdDVZci4yODlGNEE0NkRGMEEzMEQy",
      "contentDetails": {
        "videoId": "9vKqVkMQHKk",
        "videoPublishedAt": "2017-04-29T16:24:03Z"
      }
    },
    ...
  "pageInfo": {
    "totalResults": 12,
    "resultsPerPage": 50
  }
}
```

This gets us a response with all the video IDs of interest, along with a bunch of information that we don't need. To strip this out and get just the information that we care about, we can pipe this response and use jq's filter syntax:

```bash
# Hit up arrow to replace ... with previous command
$ ... | jq '.items[].contentDetails.videoId'
"WUvTyaaNkzM"
"9vKqVkMQHKk"
"S0_qX4VJhMQ"
...
```

Now we need to loop through each line in this output and call curl requesting the /videos endpoint on each record:

```bash
# `-I {}` means replace the string `{}` in the URL with the current record (video id)
$ ... | xargs -I {} curl -s "$YT_API/videos?part=snippet,statistics&id={}&key=$YT_KEY"
{
  "kind": "youtube#videoListResponse",
  "etag": "LWiYRbWgpdH_mL3I4BdCu-yKxio",
  "items": [
    {
      ...
      "snippet": {
        "publishedAt": "2017-04-28T15:58:48Z",
        "channelId": "UCYO_jab_esuFRV4b17AJtAw",
        "title": "The essence of calculus",
        ...
       }
      "statistics": {
        "viewCount": "7363949",
        "likeCount": "207221",
        "favoriteCount": "0",
        "commentCount": "6365"
      }
      ...
    }
  ],
  ...
}
{
  "kind": "youtube#videoListResponse",
  "etag": "tY3Oi8Po8BhCGIRaZNJR5NfkVWA",
  "items": [
    {
      ...
      "statistics": {
        "viewCount": "2986293",
        "likeCount": "76291",
      }
      ...
    }
  ],
  ...
}
...
```

To strip the output of unnecessary info, we can use jq's [object construction syntax](https://stedolan.github.io/jq/manual/#TypesandValues):

Notice the `|tonumber` after viewCount. This will cast the string value into a number.

```bash
$ ... | jq '{title: .items[0].snippet.title, views: .items[0].statistics.viewCount|tonumber}'
{
  "title": "The essence of calculus",
  "views": 7363968
}
{
  "title": "The paradox of the derivative | Chapter 2, Essence of calculus",
  "views": 2986300
}
...
```

Finally, we can sort this output using the `views` key and reverse it to get the results in descending order:

```bash
# With --slurp/-s, we read the entire stream of JSON objects above into a large array
# and then run the filter on the array instead of running it on each object
$ ... | jq --slurp '.|=sort_by(.views)|reverse'
[
  {
    "title": "The essence of calculus",
    "views": 7364088
  },
  {
    "title": "The other way to visualize derivatives | Chapter 12, Essence of calculus",
    "views": 3344817
  },
  ...
]
```

## tl;dr

The final command comes out to this:

```bash
$ export YT_KEY="your api key"
$ export YT_API="https://www.googleapis.com/youtube/v3"
$ export YT_LISTID="PLZHQObOWTQDMsr9K-rj53DwVRMYO3t5Yr" # Playlist id

$ curl -s "$YT_API/playlistItems?part=contentDetails&playlistId=$YT_LISTID&maxResults=50&key=$YT_KEY" | jq '.items[].contentDetails.videoId' | xargs -I {} curl -s "$YT_API/videos?part=snippet,statistics&id={}&key=$YT_KEY" | jq '{title: .items[0].snippet.title, views: .items[0].statistics.viewCount|tonumber}' | jq --slurp '.|=sort_by(.views)|reverse'
```


Sorting YouTube playlists from the command line


For some reason, YouTube doesn't let you sort playlists by criteria like views and lengths of videos.

So, I made [pl-sort](https://pl-sort.netlify.app). It allows you to enter a playlist link or id and sort it by views, duration of videos, like/dislike counts, title (alphabetically), date of publication, and of course, the default order.

It also shows you the total duration of the playlist.

For fun, I tried to make the interface as similar to YouTube's official design as I could.

If you need this frequently, you can install the [extension](https://addons.mozilla.org/en-US/firefox/addon/pl-sort) for your browser. This adds an option to open a playlist directly in pl-sort by adding a "**Open in pl-sort**" option to context menu when YouTube playlist links are right-clicked.

To contribute, submit your issues and feedback on [GitHub](https://github.com/rsapkf/pl-sort).

![pl-sort](/img/weblog/pl-sort.png)
