{"id":252,"date":"2021-12-24T08:10:00","date_gmt":"2021-12-24T08:10:00","guid":{"rendered":"http:\/\/blog.67bricks.com\/?p=252"},"modified":"2024-12-20T14:30:02","modified_gmt":"2024-12-20T14:30:02","slug":"what-have-i-been-listening-to","status":"publish","type":"post","link":"https:\/\/blog.67bricks.com\/?p=252","title":{"rendered":"What have I been listening to?"},"content":{"rendered":"\n

A while ago, Tim suggested we could have a #now-listening channel in our company Slack, in which people could post details of what they were listening to. It occurred to me that it might be a fun challenge to try to figure out from what I’d posted on there who my favourite artist was, and which was my most-listened-to album. So I rolled up my sleeves and got to work. This is an account of what I did and my various thought processes as I went along…<\/p>\n\n\n\n

Challenge: figure out how to get my posts from our #now-listening channel and do some statistics to them.<\/p>\n\n\n\n\n\n\n\n

Session 1<\/strong>: After school run, but before work…<\/p>\n\n\n\n

Start – there’s an API. https:\/\/api.slack.com<\/a><\/p>\n\n\n\n

Read the documentation: https:\/\/api.slack.com\/methods\/search.messages<\/a> looks useful – how do I call it?<\/p>\n\n\n\n

I NEED A TOKEN! Aha – https:\/\/api.slack.com\/apps<\/a> – a “generate token” button…<\/p>\n\n\n\n

Access token: xoxe.xoxp-blah-blah-blah. SUCCESS!<\/p>\n\n\n\n

First obvious question: has someone done this already? Google knows everything: https:\/\/github.com\/slack-scala-client\/slack-scala-client<\/a><\/p>\n\n\n\n

Create a new project: sbt new scala\/scala-seed.g8<\/code> – add dependency on slack-scala-client, ready to rock! In such a hurry; I can’t even be bothered to set up a package, just hijack the Hello app that came in the skeletal project…<\/p>\n\n\n\n

From docs:<\/p>\n\n\n\n

val token = \"MY TOP SECRET TOKEN\"\nimplicit val system = ActorSystem(\"slack\")\ntry {\n  val client = BlockingSlackApiClient(token)\n  client.searchMessages(WHAT TO PUT HERE?)\n} finally {\n  Await.result(system.terminate(), Duration.Inf)\n}<\/code><\/pre>\n\n\n\n
… maybe something like…?:<\/p>\n\n\n\n
val ret = client.searchMessages(\"* in:#67bricks-now-listening from:@Daniel\", sort = Some(\"timestamp\"), sortDir = Some(\"asc\"), count = Some(5))<\/code><\/pre>\n\n\n\n
RUN IT<\/p>\n\n\n\n
Fails. Because HelloSpec fails (I mentioned I just hijacked the OOTB Hello app). Fix with the delete key.<\/p>\n\n\n\n
RUN IT<\/p>\n\n\n\n
[WARN] [11\/26\/2021 08:40:37.418] [slack-akka.actor.default-dispatcher-2] [akka.actor.ActorSystemImpl(slack)] Illegal header: Illegal 'expires' header: Illegal weekday in date 1997-07-26T05:00:00: is 'Mon' but should be 'Sat'\nException in thread \"main\" slack.api.ApiError: missing_scope\nat slack.api.SlackApiClient$.$anonfun$makeApiRequest$3(SlackApiClient.scala:92)<\/pre>\n\n\n\n
:-\u200b(<\/p>\n\n\n\n
Google: “missing_scope” and interpret results<\/p>\n\n\n\n
The token used is not granted the specific scope permissions required to complete this request.<\/p><\/blockquote>\n\n\n\n
:-\u200b( :-\u200b(<\/p>\n\n\n\n
Maybe I have to create an app and add it to the workspace? I’ll try that.<\/p>\n\n\n\n
Created, figured out how to add the user token scope “search:read” – and I got a new token!<\/p>\n\n\n\n
Token= xoxp-blahblahblah<\/p>\n\n\n\n
Rerun: I got a response!<\/p>\n\n\n\n
{\n  \"ok\":true,\n  \"query\":\"* in:#67bricks-now-listening from:@Daniel\",\n  \"messages\": {\n    \"total\":0,\n    \"pagination\": {\n      \"total_count\":0,\n      \"page\":1,\n      \"per_page\":5,\n      \"page_count\":0,\n      \"first\":1,\n      \"last\":0\n    },\n    \"paging\": {\n      \"count\":5,\n      \"total\":0,\n      \"page\":1,\n      \"pages\":0\n    },\n    \"matches\": []\n  }\n}<\/code><\/pre>\n\n\n\n
:-\u200b(<\/p>\n\n\n\n
Let’s just search in the channel without specifying a name…?<\/p>\n\n\n\n
val ret = client.searchMessages(\"in:#67bricks-now-listening\", sort = Some(\"timestamp\"), sortDir = Some(\"asc\"), count = Some(5))<\/code><\/pre>\n\n\n\n
Gives:<\/p>\n\n\n\n
{\n  \"ok\": true,\n  \"query\": \"in:#67bricks-now-listening\",\n  \"messages\": {\n    \"total\": 7113,\n    \"pagination\": {\n      \"total_count\": 7113,\n      \"page\": 1,\n      \"per_page\": 5,\n      \"page_count\": 1423,\n      \"first\": 1,\n      \"last\": 5\n    },\n    \"paging\": {\n      \"count\": 5,\n      \"total\": 7113,\n      \"page\": 1,\n      \"pages\": 1423\n    },\n    \"matches\": [\n      {\n        \"username\": \"daniel.rendall\",\n        \"other\": \"field_here\"\n      }\n    ]\n  }\n}<\/code><\/pre>\n\n\n\n
Aha! My username is daniel.rendall, let’s try that:<\/p>\n\n\n\n
val ret = client.searchMessages(\"in:#67bricks-now-listening from:@daniel.rendall\", sort = Some(\"timestamp\"), sortDir = Some(\"asc\"), count = Some(5))<\/code><\/pre>\n\n\n\n
Gives:<\/p>\n\n\n\n
{\n  \"ok\": true,\n  \"query\": \"in:#67bricks-now-listening from:@daniel.rendall\",\n  \"messages\": {\n    \"total\": 3213,\n    \"pagination\": { ... etc }\n  }\n}<\/code><\/pre>\n\n\n\n
Success! Also – 3213 messages – sounds plausible. This is looking good… but sort direction seems wrong…? Try switching to “desc” => same result.<\/p>\n\n\n\n
(Time spent so far: about half an hour – better stop or will miss the morning call!)<\/p>\n\n\n\n
Session 2<\/strong>: Re-run – still works (hooray!)<\/p>\n\n\n\n
Copy and paste output and save as response.json, fix up with jq so I can examine it:<\/p>\n\n\n\n
cat response.json | jq '.' > response_tidied.json<\/code><\/pre>\n\n\n\n
And now:<\/p>\n\n\n\n
\"pagination\": {\n  \"total_count\": 3234,\n  \"page\": 1,\n  \"per_page\": 5,\n  ... etc<\/code><\/pre>\n\n\n\n
Number has gone up – I’m still listening to things!<\/p>\n\n\n\n
So, I could<\/em> parse the responses to work out what the next page should be, or I could just loop – with pages of size 100 (if the API will return them) there should be 33. So we will loop and save these as 1.json, 2.json etc. First rule of scraping – aim to do it just once and save the result locally.<\/p>\n\n\n\n
Horrible quick and dirty code alert<\/strong>!<\/p>\n\n\n\n
val outDir = new File(\"\/home\/daniel\/Scratch\/slack\/output\")\ndef main(args: Array[String]): Unit = {\n  outDir.mkdirs()\n  implicit val system = ActorSystem(\"slack\")\n  try {\n    val client = BlockingSlackApiClient(token)\n    (1 to 33).foreach { pageNum =>\n      try {\n        val ret = client.searchMessages(\"in:#67bricks-now-listening from:@daniel.rendall\",\nsort = Some(\"timestamp\"),\nsortDir = Some(\"desc\"),\ncount = Some(100),\npage = Some(pageNum))\n        Files.write(new File(outDir, \"\" + pageNum + \".json\").toPath, ret.toString().getBytes(StandardCharsets.UTF_8), StandardOpenOption.CREATE)\n        println(s\"Got page $pageNum\")\n      } catch {\n        case NonFatal(e) =>\n          println(s\"Couldn't get page $pageNum - ${e.getMessage}\")\n      }\n      Thread.sleep(1000)\n    }\n  } finally {\n    Await.result(system.terminate(), Duration.Inf)\n  }\n}<\/code><\/pre>\n\n\n\n
… prints up a reasuring list “Got page 1” => “Got page 33” and no (reported) errors!<\/p>\n\n\n\n
Second rule of scraping – having done it and got the data, zip it up and put it somewhere just in case you destroy it…<\/p>\n\n\n\n
Tidy it all (non essential, but makes it easier to look at):<\/p>\n\n\n\n
mkdir tidied\nls output | while read JSON ; do cat output\/$JSON | jq '.' > tidied\/$JSON ; done<\/code><\/pre>\n\n\n\n
On scanning the data – it looks plausible, I can’t see an obvious “date” field but there’s a cryptic “ts” field (sample value: “1638290823.124400”) which is maybe a timestamp? A problem for another day…<\/p>\n\n\n\n
(Time spent this session: about 20 minutes)<\/p>\n\n\n\n
Session 3<\/strong>: I can haz stats?<\/p>\n\n\n\n
Need to load it in. A new main method in a new object…<\/p>\n\n\n\n
val outDir = new File(\"\/home\/daniel\/Scratch\/slack\/output\")\ndef main(args: Array[String]): Unit = {\n  val jsObjects = outDir.listFiles().map { f =>\n    Json.parse(new FileInputStream(f))\n  }\n  println(jsObjects.head)\n}<\/code><\/pre>\n\n\n\n
Prints something sensible. Now need to get it in a useful form: define simplest class that could possibly work.<\/p>\n\n\n\n
case class Message(iid: UUID, ts: String, text: String, permalink: String)\nobject Message {\n  implicit val messageReads: Reads[Message] = (\n  (__ \\ \"iid\").read[UUID] and\n  (__ \\ \"ts\").read[String] and\n  (__ \\ \"text\").read[String] and\n  (__ \\ \"permalink\").read[String]\n  ) (Message.apply _)\n}<\/code><\/pre>\n\n\n\n
Not sure if I need the ID, but I like IDs. Looks like a UUID.<\/p>\n\n\n\n
… oh, also some classes to wrap the whole result with minimum of faff (and Reads, omitted for brevity):<\/p>\n\n\n\n
case class SearchResult(messages: Messages)\ncase class Messages(total: Int, matches: Seq[Message])<\/code><\/pre>\n\n\n\n
Go for broke:<\/p>\n\n\n\n
val jsObjects: Array[JsResult[Seq[Message]]] = outDir.listFiles().map { f =>\n  Json.parse(new FileInputStream(f)).validate[SearchResult].map(_.messages.matches)\n}<\/code><\/pre>\n\n\n\n
Unpleasant type signature alert – Array[JsResult[Seq[Message]]] Let’s assume nothing will go wrong and just use “.get” and “.flatMap”:<\/p>\n\n\n\n
val messages: Seq[Message] = outDir.listFiles().flatMap { f =>\n  Json.parse(new FileInputStream(f)).validate[SearchResult].map(_.messages.matches).get\n}.toList<\/code><\/pre>\n\n\n\n
That gives me 3234 Message objects, which is reassuring. They include top-level messages, and responses to threads. As far as I can see, the thread responses include a ?thread_ts parameter in their permalink, therefore filter them out – leaves 1792 remaining.<\/p>\n\n\n\n
val filtered = messages.filterNot(_.permalink.contains(\"?thread_ts\"))\nfiltered.take(10).map(_.text).foreach(println)<\/code><\/pre>\n\n\n\n
…and voila:<\/p>\n\n\n\n