Grab a section from a large file between two known substrings using sed?

I have a pretty large XML file without line-breaks.

It's so big it's slow to open and operate on in Emacs or other text editor. But I just want to extract a shortish section of it between two known substrings.

I don't care about preserving the XML structure, I just want a chunk of characters.

This ought to be a one-liner in sed, no?

Any idea how to do this? I tried adapting but it doesn't seem to work when I pipe my file into it. (It works on toy examples, but I'm thinking that my file may be too big.)

2

3 Answers

With GNU grep:

With frompattern and topattern in output:

grep -o 'frompattern.*topattern' file.xml

Without frompattern and topattern in output:

grep -Po 'frompattern\K.*(?=topattern)' file.xml

2

Well, usually it's easy to do with sed. But it's ALWAYS easy to do it with awk:

awk '/frompattern/,/topattern/' your.xml > chunk.xml

Here the two patterns are regular expressions (as would be with sed). If it discourages you for any reason, you can use simple strings, if you know where they are:

awk '$x=="fromstring",$y=="tostring"' your.xml > chunk.xml

Here x and y are the field positions of the strings you want to be the barrier signs. (More can be done with tiny effort.)

2

I use a command called pv (Pipe viewer) - once installed you can see the progress of the command and how long to complete. great for large files

For Mac - (for others go to ostechnix.com)

brew install pv

my example is using sed.

pv bigFile.txt | sed -n '/^FIRST PART OF STRING/,/^LAST PART OF STRING `/p' > output.txt

If you dont use pv you can just use anything else to echo the file

eg. cat or echo

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password Submit

Post as a guest

Post Your Answer Discard

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy