View source for Perl

{{cleanup}}
<b>P</b>ractical <b>E</b>xtraction and <b>R</b>eport <b>L</b>anguage is the oldest of the [[interpreted languages]], [[python]] being its 3 years younger sibling.  The perl interpreter is written in [[C]], a [[compiled languages|compiled language]].

Perl is flexible and can be used to write [[web applications]], command line applications, or services.

==Development Environment==
To develop in perl you will need only a perl interpreter and a text editor.  For those of you who find un highlighted perl, there are a variety of windows & linux text editors with syntax highlighting support.

<b>Windows:</b>
* notepad++
* cygwin's vim implementation

<b>Linux:</b>
* vim
* nano
* emacs
* geany
* gedit

===Linux & Unix===
On most distributions, perl and cpan come bundled by default.  In the case it is not, a simple apt-get, emerge, yum install, pacman, or any other package manager should install it quickly.  You can determine if perl is installed by typing `which perl' at the bash command line.  If a filename is returned, you're good to go.

===Windows===
You can do everything we're going over by installing cygwin with perl. CYGWIN is available at http://www.cygwin.com/install.html
  For compilation to .exe, we recommend "pp", you can install this by `typing cpan -i pp' from your cygwin shell.
{{info|There is also a perl written by activestate- searching for "activestate perl" in any search engine will find it}}

===CPAN===
CPAN is the module and package installer for perl.  It can be accessed on most distributions simply by typing `cpan'.  On windows, you can access it by typing `cpan' in your CYGWIN shell.
{{notice|If `cpan' does not work, try `perl -e 'shell' -mCPAN'.  If this does not work, your installation may be broken.}}

==Your first program==
===Code===
To run this code, you'll only need to put it in a text file.  Save it as "hello.pl", and then you can execute the following to run it from either cygwin or [[bash]]:
* chmod +x hello.pl
* ./hello.pl

Alternatively you can simply type:
* perl hello.pl

{{code|text=<syntaxhighlight lang="perl">#!/usr/bin/perl
use strict;
use warnings;
print "Hello world!\n";</syntaxhighlight>}}

===Analysis===
The shebang declares the location of the code's interpreter. I.e. if you're writing bash, you'll need to put:

  #!/bin/bash

at the top of your file. In perl, it's typically:

  #!/usr/bin/perl

This should be the first line in any perl you write. You can also use:

  #!env perl

If you are unsure of the path and you have it in your environment variables. 
{{notice|If for some reason `#!env perl' and `#!/usr/bin/perl' do not work, running `which perl' from the [[bash]] command line will return the proper bath.}}

This is only required if you want to directly execute your script (i.e. ./script.pl).  If you get permissions errors when attempting this, you can execute it via `perl script.pl' or running `chmod +x script.pl' before running `./script.pl'.  

With perl in particular, its real easy for ugliness to occur. To counter this, the next lines are:

{{code|text=<syntaxhighlight lang="perl">use strict;
use warnings;</syntaxhighlight>}}

Strict perl forces you to maintain some semblence of syntax. Without the strict usage, you can basically run amok with code, perl will not care.

The print "Hello world!\n" line simply prints "Hello world!" with a '''newline''' character on the end.  On windows you may need to change "\n" to "\r\n", depending on which interpreter you've installed.  

{{info|You can also reference the hex code for this via a \x character, "\x0a\x0d".}}

==Variables & Data Types==
In strict perl, variables must be declared using the "my" or "our" operators.  "my" is used implicitly in non-shared memory, whereas "our" is used explicitly for shared memory to pass data between threads.

===Scalars===
Scalars in perl are prefixed with a '''$'''.  A scalar may contain any string, integer, or floating point value.  It may also contain a reference pointer. An example declaration:
{{code|text=<syntaxhighlight lang="perl">my $message = "Hello world!\n";
print $message;</syntaxhighlight>}}

===Arrays===
Arrays (or lists) have elements.  Typically an array in perl can contain anything - each element can be something different.  An array element may be a hash, hash reference, scalar, or another array.

Arrays are prefixed by the '''@''' character:
{{code|text=<syntaxhighlight lang="perl">my @messages = ("Hello world!\n","I like perl!\n");
print $messages[0];
print $messages[1];
print "Size of messages array: ". #$messages . "\n";
</syntaxhighlight>}}

{{info|You can access and modify array elements directly:}}
{{code|text=<syntaxhighlight lang="perl">
$messages[0] = "Hello world!\n";
</syntaxhighlight>}}

====Helper Functions====
=====join()=====
Join will compile an array into a scalar.  Using the array example above, '''@messages''', the following code will generate the string "Hello world!\n, I like perl!\n" as a scalar:
{{code|text=<syntaxhighlight lang="perl">my @messages = ("Hello world!\n","I like perl!\n");
my $joined_message = join(", ",@messages);
print $joined_message;</syntaxhighlight>}}

=====split()=====
Split takes a scalar and converts it to an array using a delimiter.  Using our string from earlier:
{{code|text=<syntaxhighlight lang="perl">my $joined_message = "Hello world!\n, I like perl!\n";
my @messages = split('/, /',$joined_message);
print $messages[0];
print $messages[1];
print "Size of messages array: ". $#messages . "\n";</syntaxhighlight>}}

=====push()=====
{{info|The '''push()''' function is used to append an element or elements to the end of an array.}}{{code|text=<source lang="perl">
my @array;
push(@array,'element one');
push(@array,('element two','element three'));
</source>}}{{quote|You can also add to the end of an array with: <source lang="perl">$array[$#array] = "new element";</source>|The professor}}

=====pop()=====

=====unshift()=====

=====shift()=====

===Hashes===
A hash is very similar to a struct in [[C]].
====Introduction====
Hashes are prefixed by the '''%''' character.  Hash element values are prefixed by '''$'''.  A hash element may contain another hash, an array, or a scalar.

{{info|You can directly modify the key inside of a hash:}}
{{code|text=<syntaxhighlight lang="perl">$hash{'key'} = 'value';</syntaxhighlight>}}

{{info|You can also create a key &#61;> value pair on declaration:}}
{{code|text=<syntaxhighlight lang="perl">my %hash = ('key' => 'value', 'key2' => 'value2');</syntaxhighlight>}}

* Example:
{{code|text=<syntaxhighlight lang="perl">my %user;
$user{'username'} = "hatter";
$user{'network'}  = "irc.blackhatacademy.org";
print "The user " . $user{'username'} . " is connected to " . $user{'network'} . "\n";
</syntaxhighlight>}}

====Helper Functions====
=====each()=====
{{notice|This uses a while loop.  While loops are covered later in this article.}}{{info|"'''while my each'''" can be used to isolate $key &#61;> $value pairs from a hash as follows with our %user hash:}}
{{code|text=<syntaxhighlight lang="perl">while(my($key,$value) = each(%user)) { print "Key: $key, Value: $value\n"; };</syntaxhighlight>}}

=====keys=====
{{notice|This uses a foreach() loop and casting.  Foreach and casting are covered later in this article.}}{{info|We can isolate $key&#61;>$value pairs the same as above using keys in stead of each:}}
{{code|text=<syntaxhighlight lang="perl">foreach my $key (@{sort keys %user}) { print "Key: $key, Value: ". $user{$key} ."\n"; };</syntaxhighlight>}}

===References===
{{info|A reference is very similar to a pointer in [[C]].}}
====Hash References====
{{info|A hash reference is a scalar created using the '''\ '''operator as follows:}}
{{code|text=<source lang="perl">
my %user;
$user{'name'}    = "hatter";
$user{'network'} = "irc.blackhatacademy.org";

my $hashref = \%user;
</source>}}
{{notice|Once you've created a hashref (hash reference) you must use '''pointers''' to access a key:}}
{{code|text=<source lang="perl">
print $user->{'name'} . "\n";
print $user->{'network'} . "\n";
</source>}}

====Callback References====
{{notice|This involves user-defined functions.  User-defined functions are covered later in this article.}}
A callback reference is a scalar that points to a function.  To create a callback reference:
{{code|text=<source lang="perl">my $callback = \&function_name;
</source>}}
To execute the callback function and pass it arguments:
{{code|text=<source lang="perl">$callback->($arg1, $arg2);</source>}}

===Casting===
{{info|Casting is the process of transitioning from one data type to another.}}
This is typically done using curly brackets '''{}''' preceeded by a data type designator ($,%, or @).  

* '''To cast a hash reference back to a hash:'''
{{code|text=<source lang="perl">
my %hash;
my $hashref = \%hash; #create the hash reference

my %casted  = %{$hashref}; #Cast back to a hash.
</source>}}
* '''To cast a list of keys in a hash into an array:'''
{{code|text=<source lang="perl">
my @casted = @{keys %hash};
</source>}}
* '''To cast a scalar value to an integer:'''
{{code|text=<source lang="perl">
my $integer = int($scalar);
</source>}}

==Boolean Logic==
===Operators===
====Mathematical====
* '''='''
The '''=''' operator assigns a value to a variable.

* '''=='''
The '''==''' operator is used to test if a variable is equal to a value or another variable.

* '''!'''
The '''!''' operator means "not".  It applies to '''=''' and any variable.  For example, "'''if (!$scalar)'''" is used to determine if the '''$scalar''' variable is null, zero, or undefined.  When used before a '''=''' operator, this becomes "not equal to".

* '''eq'''
Sometimes because variables may not be a string or integer, the '''eq''' operator is used to determine if the two are equal.

{{info|The following operators are used for greater than, less than, and greater than or equal to, less than or equal to, etc; similar to other languages:}}
* '''>'''
* '''gt'''
* '''<'''
* '''lt'''
* '''>='''
* '''gte'''
* '''<='''
* '''lte'''

====Regular Expression====
{{info|The '''~''' operator is used with regular expressions, which are covered later in this article.}}
The ~ operator can be used in a variety of ways:
* '''=~'''
* '''!~'''

===Statements===
====if====
{{info|An '''if''' statement may have 3 types of clauses: '''if''','''elsif''', and '''else'''.}}
For the below example, assume that the $age scalar is passed as a command line argument:
{{code|text=<source lang="perl">
if (int($age) == $age) {  #Making sure it's an integer.
    if ($age < 18) {      #If the age is less than 18:
        print "You must be at least 18 to view this.\n";
    } elsif ($age < 21) { # If the age is more than 18, but less than 21:
        print "Because you are under 21, some features may be restricted.\n";
    } else {              # If none of the conditions have been met:
        display_content();
    }
}
</source>}}

====unless====
{{info|An '''unless''' statement may only have the '''unless''' clause and an '''else''' clause.}}
{{code|text=<source lang="perl">unless ($age >= 21) {
    print "All content is restricted for users under the age of 21.\n";
} else {
    print "Welcome to our sample age gate!\n";
    display_content();
}
</source>}}

====AND an OR====
{{info|And an or are used to apply multiple conditions to a boolean statement.}}
* '''&&''' is the way perl represents "and"
* '''||''' is the way perl represents "or"
Example:
{{code|text=<source lang="perl">
  if ($age < 21 && $age >= 18) {
      print "Some content will be restricted because you are not older than 21.\n";
  }
</source>}}

====switch====
{{info|To use perl's '''switch()''' routine you must have '''use Switch;''' before your switch() statement.}}
A switch statement allows a programmer to avoid long chains of "elsif" statements.  It condenses the amount of required lines of code.  Perl's switch statement is very similar to the switch() statement in [[C]] and [[C++]], though the syntax is a little different.
{{info|A perl switch() statement may contain case and else clauses.}}
Perl switch cases can also be used to determine if a value is in a list, an array element, hash key, or matches a regular expression or string.{{notice|In this example, suppose '''$option''' was a numeric value for an integer based menu with 3 options.}}
{{code|text=<source lang="perl">
use Switch;
switch(int($option)) {
    case 1 {  # Essentially the same as if ($option == 1)
        print "You picked option 1!\n";
    }
    case 2 { # Essentially the same as elsif ($option == 2)
        print "You picked option 2!\n";
    }
    case 3 { # Essentially the same as elsif ($option == 3)
        print "You picked option 3!\n";
    }
    else { 
        print "invalid menu option!\n";
    }
}
</source>}}
For more information, see perldoc switch, or : http://perldoc.perl.org/Switch.html

====Golfing====
{{info|The term '''golfing''' applies to condensing a boolean statement into one line.}}
{{quote|Golfing is typically used when you only need to execute one line of code for a boolean statement.|The professor}}
{{code|text=<source lang="perl">
print "You are not 18 or older!\n" unless ($age >= 18);
</source>}}
* Is essentially the same as:
{{code|text=<source lang="perl">
print "You are not 18 or older!\n" if ($age < 18);
</source>}}
* Is essentially the same as this '''un-golfed''' statement:
{{code|text=<source lang="perl">
unless ($age >= 18) {
    print "You are not 18 or older!\n";
}
</source>}}

===Helper Natives===
{{info|These helper natives are '''boolean''' statements that assist with the determination of the existence of or the defining of a variable.}}
====exists====
The '''exists''' native applies specifically to hashes and hash references.  
{{code|text=<source lang="perl">print "This user has an age.\n" if exists $user->{'age'};
</source>}}

====defined====
The '''defined''' native determines if a '''scalar''' value is defined:
{{code|text=<source lang="perl">
print "We received a response from the server.\n" if defined $response;
</source>}}

====undef====
The '''undef''' native determines if a '''scalar''' value is un-defined:
{{code|text=<source lang="perl">
print "We received a response from the server.\n" unless undef $response;
</source>}}

===Bitwise Manipulations===
{{info|Perl's bitwise manipulations cover the syntax for performing '''[[Bitwise Math|bitwise math]]''' on variables.}}
====[[Bitwise_Math#3.2_-_AND|AND]]====
* '''&''' - The '''AND''' operator.
{{code|text=<source lang="perl">
my $num = 10;
$num = $num & 25;
print $num . "\n";
</source>}}
====[[Bitwise_Math#3.1_-_NOT|NOT]]====
* '''~''' - The '''NOT''' operator
{{code|text=<source lang="perl">
my $num = 10;
$num = ~$num;
print $num . "\n";
</source>}}
====[[Bitwise_Math#3.3_-_OR|OR]]====
* '''|''' - The '''OR''' operator
{{code|text=<source lang="perl">
my $num = 10;
$num = $num | 25;
print $num . "\n";
</source>}}
====[[Bitwise_Math#3.4_-_XOR|XOR]]====
* '''^'''
{{code|text=<source lang="perl">
my $num = 10;
$num = $num ^ 25;
print $num . "\n";
</source>}}
====[[Bitwise_Math#1.1.0_-_Logical_Shifts|Bit Shifting]]====
* '''<<''' - The '''shift left''' operator
* '''>>''' - The '''shift right''' operator
{{code|text=<syntaxhighlight lang="perl">my $num = 10;
$num = $num << 2; #Shift left two bits
$num = $num >> 2; #Shift right two bits
print $num . "\n";
</syntaxhighlight>}}

====[[Bitwise_Math#2.0_-_Circular_Shift_or_Bit_Rotation|Bit Rotation]]====
{{info|Perl bit rotation requires the [http://search.cpan.org/~stbey/Bit-ShiftReg-2.0/ShiftReg.pm Bit::ShiftReg package from CPAN].  More information available there.}}

==Loops==
===While===
===Until===
===For===
===Foreach===

==User Input==
===Command Line Arguments===
====GetOpt::Std====
====GetOpt::Long====

===STDIN (Standard Input)===
===HTTP Inputs===

==Regular Expressions==



==LESSON==

Perl Regex - Hatter

1.0 - Introduction
The shebang declares the location of the code's interpreter. I.e. if you're writing bash, you'll need to put:

<syntaxhighlight lang="bash">#!/bin/bash</syntaxhighlight>

at the top of your file. In perl, it's typically:

<syntaxhighlight lang="bash">#!/usr/bin/perl</syntaxhighlight>

This should be the first line in any perl you write. You can also use:

<syntaxhighlight lang="bash">#!env perl</syntaxhighlight>

If you are unsure of the path and you have it in your environment variables. With perl in particular, its real easy to get it ugly as hell.

To counter this, your next two lines will be:

<syntaxhighlight lang="perl">
use strict;
use warnings;
</syntaxhighlight>

Strict perl forces you to maintain some semblence of syntax. Without the strict usage, you can basically run amok with code, perl does not care.

Our script so far should look like:

<syntaxhighlight lang="perl">
#!/usr/bin/perl
use strict;
use warnings;
</syntaxhighlight>

Now, thanks to mepholic, I've removed sensitive data from this log and made it available for everyone:

  http://blackhatacademy.org/sample_asterisk.log

You will need to save this file.

WINDOWS USERS: 

TODAY we'll be writing a regex to read asterisk logs to determine ip's with authentication failures.

Our perl script is going to do the following :

1) Analyze an asterisk log
2) Determine IP's with auth failures
3) Determine number of auth failures per IP
4) Block IP's going over a configurable threshhold

To participate in this lesson, please download and install perl and a text editor of your choice.

You've got a few different data types in perl and its different than other languages.
You have scalars (ints or strings) prefixed with $
You have arrays/lists prefixed with @
You have hashes (similar to a struct/associative array) prefixed with %
You have references prefixed with a \

You can declare a variable by saying:

<syntaxhighlight lang="perl">
my [$%@]varname;
</syntaxhighlight>

So if its a scalar:

<syntaxhighlight lang="perl">
my $scalar;
</syntaxhighlight>

So :

<syntaxhighlight lang="perl">
my $scalar = 0;
my $scalar = "foo";
</syntaxhighlight>

Either is acceptable or just:

<syntaxhighlight lang="perl">
my $scalar;
</syntaxhighlight>

Now, for our example, we'll want to analyze each line. So let's look at these failed lines:

  [Aug 27 07:34:43] NOTICE[2762] chan_sip.c: Registration from '"1561"<sip:1561@0.0.0.0>' failed for '113.105.152.180:40586' - No matching peer found

In this particular instance, we know the ip comes right before the last : in the string.
So to match the ip itself and not the whole line, We'd do something like this:

  /\x27([\d]{1,3}\.[\d]{1,3}\.[\d]{1,3}\.[\d]{1,3}):[^:]+$/

Where \x27 is a single quote. Obviously, this won't work for failed auths because it will match shit that isn't a failed auth. It'll also match successes or any line with an ip like that.

Here comes an in-depth explanation of that line:

Single quote, any digit between one and 3 characters in length, a dot, so on and so forth (match an IP), until the last : in the string, and it has to be right before the last : in the string.
The [^:]+ at the end makes it so if a : occurs after that, it will no longer match as [^:]+ matches everything until the end of the string.

Now we've got to make it match failures too, the only real important data we care about is the ip, the count of failed auths and whether or not they're already in iptables for block because if they are we don't want to add another rule.

It means we've already blocked'em and these were previously analyzed logs

We'll use a hash to hold our ip's with failed auth and the value of each $hash{$ip} will contain the count of the failures. %hash is how it is declared and referred to but when you refer to a key of the hash, you want to put $ since its a scalar value you're accessing inside the hash.

You can directly modify the key inside of a hash by doing:

<syntaxhighlight lang="perl">
$hash{'key'} = 'value';
</syntaxhighlight>

Also, you can create an key=>value pair by doing:

<syntaxhighlight lang="perl">
my %hash = ( 'key' => 'value', 'key2' => 'value2' );
</syntaxhighlight>

You can copy the hash tree to another hash by doing:

<syntaxhighlight lang="perl">
my %hash2 = %hash;
</syntaxhighlight>

If you want to print out a tree of the hashes, you can use:

<syntaxhighlight lang="perl">
while(my($key,$value) = each(%hash)) { print "Key: $key, Value: $value\n"; };
</syntaxhighlight>

Resulting Log:

  phobos public_html # perl test.pl
  113.105.152.180:10001
  220.134.238.64:2194
  67.205.85.58:7789
  80.254.76.242:4

So, http://blackhatacademy.org/test.pl.txt, will spit out your ip's and number of failures. This data is then used to blackhole IP's (preventing them from affecting the server)

That'll add a blackholed IP route if 

1) They are over $threshhold failures (25)
2) They are not already blackholed