Class: TyrantManager::Commands::ArchiveUlogs

Inherits:
TyrantManager::Command show all
Defined in:
lib/tyrant_manager/commands/archive_ulogs.rb

Overview

Archive ulog files that are no longer used for replication.

You must give the list of slave connection strings on the commandline, or if the server is in a single master-master relationship, then you can allow tyrantmanager to figure that out and do the right thing.

Give the slaves of a particular master

tyrantmanager archive-ulogs master1 --slaves slave1:11011,slave2:11012,slave3:11013

Or if there is a master-master relationship tyrant manager can figure it out.

tyrantmanager archive-ulogs master1

Sometimes it is useful to force a record from the master to the slave(s) so that the replication timestamp (rts) in the slaves is updated.

This is mainly useful for archiving the ulog files in the failover master of a master-master setup. The failover may never have an updated ‘rts’ value since its slave is the primary master, and the primary master may not have replicated from the failover master in quite some time.

tyrantemanager archive-ulogs --force

Instance Attribute Summary

Attributes inherited from TyrantManager::Command

#manager, #options

Class Method Summary collapse

Instance Method Summary collapse

Methods inherited from TyrantManager::Command

#after, #before, #command_name, #error, find, inherited, #initialize, list, #logger

Constructor Details

This class inherits a constructor from TyrantManager::Command

Class Method Details

.command_name ⇒ Object 



38
39
40
 
# File 'lib/tyrant_manager/commands/archive_ulogs.rb', line 38

def self.command_name
  'archive-ulogs'
end

Instance Method Details

#archive_files_with_method(list, method) ⇒ Object

Archive each of the files in the list with the given method



80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
 
# File 'lib/tyrant_manager/commands/archive_ulogs.rb', line 80

def archive_files_with_method( list, method )
  list.each do |fname|
    leader = (options['dry-run']) ? "(dry-run)" : ""
    if "delete" == method then
      File.unlink( fname ) unless options['dry-run']
      manager.logger.info "deleted #{leader} #{fname}"
    else
      if File.extname( fname ) == ".ulog" then
        %x[ gzip #{fname} ] unless options['dry-run']
        manager.logger.info "#{leader} compressed #{fname}"
      else
        manager.logger.debug "#{leader} already compressed #{fname}"
      end
    end
  end
end

#determine_master_slave_mappings(list_of_slaves) ⇒ Object

Given a list of slaves, create a hash that has as the key, the instance connection string of a master, as as the value, an array of Rufus::Tokyo::Tyrant connections.



163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
 
# File 'lib/tyrant_manager/commands/archive_ulogs.rb', line 163

def determine_master_slave_mappings( list_of_slaves ) 
  master_to_slaves = Hash.new{ |h,k| h[k] = [] }

  list_of_slaves ||= []
  list_of_slaves.each do |connection_string|
    host, port = connection_string.split(":")
    conn = Rufus::Tokyo::Tyrant.new( host, port.to_i )
    stat = conn.stat
    master_host = stat['mhost']
    master_port = stat['mport']
    if master_host and master_port then
      master_to_slaves["#{master_host}:#{master_port}"] << conn
    else
      manager.logger.warn "Slave #{connection_string} does not have a master server configured, skipping using it as a slave"
    end
  end

  return master_to_slaves
end

#potential_ulog_archive_files(instance) ⇒ Object

The list of potential files to remove from the ulog directory. This is all the files in the ulog directory, minus the last one lexically.

A hash of filename => File.mtime values is returned



148
149
150
151
152
153
154
155
156
 
# File 'lib/tyrant_manager/commands/archive_ulogs.rb', line 148

def potential_ulog_archive_files( instance )
  names = instance.ulog_files[0..-2]
  names += Dir.glob( "#{instance.ulog_dir}/*.ulog.gz" )
  potential = {}
  names.each do |name|
    potential[name] = File.mtime( name )
  end
  return potential
end

#remove_newer_than(potential, whence) ⇒ Object

Filter all the ulogs from the list whose value is greater than the input time.



131
132
133
134
135
136
137
138
139
140
 
# File 'lib/tyrant_manager/commands/archive_ulogs.rb', line 131

def remove_newer_than( potential, whence )
  fmt = "%Y-%m-%d %H:%M:%S"
  potential.keys.sort.each do |fname|
    if potential[fname] > whence then
      manager.logger.info " ulog #{fname} timestamp #{potential[fname].strftime(fmt)} is newer than #{whence.strftime(fmt)}. Skipping."
      potential.delete( fname )
    end
  end
  return potential
end

#run ⇒ Object 



42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
 
# File 'lib/tyrant_manager/commands/archive_ulogs.rb', line 42

def run
  instance_slaves = determine_master_slave_mappings( options['slaves'] )
  current_timestamp = Time.now.strftime("%Y-%m-%d %H:%M:%S")

  manager.each_instance( options['instances'] ) do |instance|
    if instance.is_master_master? then
      manager.logger.debug "#{instance.name} is master_master"
      instance_slaves[instance.connection_string] << instance.master_connection

      if options['force'] then
        msg = "Forcing a record #{options['force-key']} => #{current_timestamp} from master #{instance.connection_string} through to slaves"
        msg = "(dry-run) #{msg}" if options['dry-run']
        manager.logger.info msg
        if not options['dry-run'] then
          c = 
          instance.connection[options['force-key']] = current_timestamp
        end
      end
    end

    slave_connections = instance_slaves[instance.connection_string]

    potential_removals = potential_ulog_archive_files( instance )
    manager.logger.info "Checking #{slave_connections.size} slaves of #{instance.name}"

    slave_connections.each do |conn|
      manager.logger.info "Slave has #{options['force-key']} => #{conn[options['force-key']]}" if options['force']
      potential_removals = trim_potential_removals( instance, potential_removals, conn )
    end

    archive_files_with_method( potential_removals.keys.sort, options['archive-method'] )

  end
end

#trim_potential_removals(master_instance, potential, slave_conn) ⇒ Object

Remove all items from the potential removals list that have a timestamp that is newer than the last replication timestamp of the tyrant connection



102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
 
# File 'lib/tyrant_manager/commands/archive_ulogs.rb', line 102

def trim_potential_removals( master_instance, potential, slave_conn )
  slave_stat = slave_conn.stat
  milliseconds = Float( slave_stat['rts'] )
  last_replication_at = Time.at( milliseconds  / 1_000_000 )

  manager.logger.info "Slave #{slave_conn.host}:#{slave_conn.port} last replicated at #{last_replication_at.strftime("%Y-%m-%d %H:%M:%S")} from #{master_instance.connection_string}"

  if milliseconds.to_i == 0 then
    manager.logger.warn "It appears that the slave at #{slave_conn.host}:#{slave_conn.port} has never replicated"
    manager.logger.warn "This means that no ulogs on the master will be removed, since there is no evidence of replication."
    manager.logger.warn "It may be necessary to force a replication by inserting a record into the master:"
    manager.logger.warn ""
    manager.logger.warn "    tcrmgr put -port #{master_instance.configuration.port} #{master_instance.configuration.host} __tyrant_manager.force_replication_at #{Time.now.utc.to_i}"
    manager.logger.warn ""
    manager.logger.warn "And then removing that record if you so choose:"
    manager.logger.warn ""
    manager.logger.warn "    tcrmgr out -port #{master_instance.configuration.port} #{master_instance.configuration.host} __tyrant_manager.force_replication_at"
    manager.logger.warn ""
  end

  manager.logger.debug "Trimming ulog files that are newer than #{last_replication_at.strftime("%Y-%m-%d %H:%M:%S")}"

  trimmed_potential = remove_newer_than( potential, last_replication_at )
end