searching for "created_at": followed in the next line by "retweeted": in lots of json files for a specific range

Question

I have a large amount of json files and previously I was told to use the following line to search if a tweet is created at a specific time range in a date:

grep -wirnEzc '},.*created_at":\s"Wed Oct 19 2(1:[0-5][0-9]:[0-5][0-9]|2:([0-2][0-9]:[0-5][0-9]|30:00)) .* 2016' *

I am not aware what are the following exactly trying to represent:

2(1:[0-5][0-9]:[0-5][0-9]|2:([0-2][0-9]:[0-5][0-9]|30:00))

As for now, I need to change it so that I can find the number of results returned by this grep that are between between July 17, 2016 and November 8, 2016 for the "created_at" field. Here's an example of one of those json files: https://hastebin.com/budovutume.scala

As you see each json file can contain various tweets. So basically I am looking to search through all these *.json files and search for the "created_at": patterns that are followed in the next line by "retweeted": because other things like profile or profile image could have the attribute "created_at":. And the date range I am looking for is July 17 2016 to November 8 2016.

An example of interest (not date-wise):

}, 
            "created_at": "Wed Dec 14 22:34:28 +0000 2016", 
            "retweeted": false, 
            "coordinates": null, 
            "in_reply_to_user_id_str": null, 
            "source": "<a href=\"https://ifttt.com\" rel=\"nofollow\">IFTTT</a>", 
            "in_reply_to_status_id_str": null, 
            "in_reply_to_screen_name": null, 
            "in_reply_to_user_id": null, 
            "extended_entities": {

This is not Ubuntu-specific, so it would be better asked on Stack Overflow. — wjandrea, Apr 05 '18 at 01:03
In your previous (linked) question, it was also suggested that you should treat this as a structured data processing task rather than a text processing task and in particular to use `jq`. If you do that, then I think you should be able to distinguish the `created_on` attribute of the top level object and that of its descendants i.e. `jq '.[] | {created_at}' file.json` versus (for example) `jq '.[] | .user | {created_at}' file.json` directly, without relying on other textual attributes. — steeldriver, Apr 05 '18 at 02:00
You *might* be able to use a `select` to choose a date range as well - although you may need to get the dates into ISO format in order to make them lexically sortable. See for example [jq between two dates/days](https://github.com/stedolan/jq/issues/1056) — steeldriver, Apr 05 '18 at 02:02

score 0 · Answer 1 · answered Nov 24 '18 at 03:12

To answer your first question: the expression 2(1:[0-5][0-9]:[0-5][0-9]|2:([0-2][0-9]:[0-5][0-9]|30:00)) is simply a regular expression, matching strings that start with a 2, followed either by 1:AB:CD, where A and C are digits from 0-5 and B and D are digits from 0-9, or by 2:AB:CD, where A is a digit from 0-2, C is a digit from 0-5, and B and D are digits from 0-9, or by 2:30:00. This is a very crude way to match all times between 21:00:00 and 22:30:00, inclusive. Writing such a regular expression to match dates or times between some arbitrary starting point and endpoint would be a very tedious exercise.

The link to your JSON file is no longer active, but I'll try to answer your main question based on the information you provided. By the way, your original grep command is a bit contradictory, since you give both the option -n (print line numbers) and -c (print only the count of the number of lines matching), which are mutually exclusive. So I am not exactly sure what information you want to extract (line numbers or count).

Here is a command that will extract all the lines containing the words created_at and which precede a line containing the words retweeted:

grep -rn -B1 'retweeted' * | grep 'created_at'

The idea is that the first grep prints each line matching retweeted and also the line before it. The second grep filters out the lines containing created_at.

Now you'll have a list of all the dates, with their line numbers. Filtering out the date range is harder; if you only have to do this rarely, you could use the regular expression

grep -E '(July (1[7-9]|[23].)|August .*|September .*|October .*|November [1-8]), 2016'

for your particular date range (but the more complicated the range, the harder this method gets).

searching for "created_at": followed in the next line by "retweeted": in lots of json files for a specific range

1 Answers1